- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents Arxidea, a human-in-the-loop agentic system that takes one researcher's publication record and one week of arXiv postings and produces checked research ideas. It has four stages, coordinated by orchestrator agents through a file-based memory.
Critical strengths
-
A fully graded, researcher-specific evaluation pool, used carefully. Most personalization studies in scholarly recommendation rely on clicks, citation proxies or simulated users. Here the profiled researcher graded every paper in a week-sized uniform sample (2,574 papers), including those no method retrieved. This allows real recall to be computed and every design to be compared on equal terms (l. 237–248). The analysis is careful:
- selection among 1,799 configurations uses a fixed rule and is reported cross-fitted across odd and even weeks (l. 305–312);
- bootstrap intervals are paired;
- profile builds are repeated five times;
- grade consistency is checked with 90 duplicates.
The resulting negative finding is informative for practitioners. A cross-encoder and two LLM rerankers each make the list worse, replacing about 40 relevant papers with 21–24 relevant ones (l. 369–377, Table 3), and LLM runs are unstable (66–70 of 100 picks shared). Releasing the pool and grades would be a useful resource.
-
A design that records evidence and separates roles. The system is organized so that no agent judges its own work:
- numbers enter the specification only through a program-maintained registry;
- a model-free program replays tests and searches before a verdict is recorded (l. 208–210, 225–226);
- eliminated ideas keep their recorded reasons.
Given documented failure modes of autonomous research agents (model judges rating weak proposals favorably, fabricated data, component substitution; l. 100–103), this is a sensible architecture, and the three runs show it can return negative and flagged verdicts, not only passes.
Critical weaknesses
-
The paper's main claims, personalized ideation and evidence-backed verification, are not evaluated in a way that supports them.
- The ideas are never assessed by the researcher. The abstract criticizes prior work for evaluating personalization "using simulated users or model-based judges rather than the intended researcher" (l. 16–18). Yet the ideas are evaluated only by a model-based judge (GPT-6 Sol; §5.1, App. E). R1 graded all 2,574 papers but none of the 79 ideas. There is no evidence that the ideas are personalized (relevant to R1's threads), novel or useful to R1, which is what the title claims.
- The Desk Check evaluation is partly circular. The grader's case files include "registered premise checks, executed test outputs, a written derivation, and the closest existing work found by literature search" (l. 1118–1120). This is the same evidence on which the Desk Check based its decisions. Removing only the explicit verdict field does not make the grader independent: if a premise test failed or close prior work was found, both the Desk Check and the grader will downgrade the idea. The 0.79-point gap may therefore show consistency between two readers of the same evidence rather than correct filtering. A grader that sees only the idea text (and seed abstract), or a human expert blind to the evidence, is needed.
- No baselines and weak filtering. There is no comparison with simpler filters: a single-call LLM critique, self-refinement, random elimination at the same rate, or the system without the replay program. Even under the model grader, 40% of retained ideas are Fatal or Flawed and only 13% are Good or better (Fig. 5). False negatives (good ideas eliminated, 18% graded Fair or better) are not analyzed.
- The Experiment Check is shown, not evaluated. It is supported by one review round from each of three runs (§5.2, App. F), described in paraphrase. There is no measure of whether pass, fail or flag verdicts are correct, how often the auditors catch numerical errors, or how reviewer votes relate to expert judgment. The cost is also substantial: 203M and 748M tokens for the fail and flag runs (Table 10), with no comparison to the researcher's own time.
- Missing related work on idea evaluation. The paper does not engage with large-scale human evaluations of LLM-generated research ideas, e.g., Si et al. (ICLR 2025, 100+ NLP researchers judging novelty and feasibility), or with ideation systems that ground ideas in retrieved literature, e.g., ResearchAgent (Baek et al., 2025), Scideator and Chain of Ideas. These are the natural baselines and evaluation protocols for §5.
-
The recommendation evaluation rests on a single researcher, and parts of it are built into the grades.
- N = 1. All configuration selection and the headline numbers (84/100, the reranker result) come from R1's grades. R1 appears closely tied to the project: R1 confirmed the profile, answered Experiment Check points and asked for four papers to be omitted from the judge's copy (l. 999–1000, 506). If R1 is an author, the grades are not independent of system design. The paper should state this.
- Some grades are assigned by a category rule that matches a deployed signal. 380 of the 2,574 grades (15%) are set to 0 by a rule based on arXiv primary categories (l. 746–750). The deployed category signal ranks papers by the same arXiv categories (l. 170–171). This builds part of the category signal's success into the ground truth. It may also explain why the category signal is among the most useful additions (Fig. 3) and why random baselines are low.
- The comparison with the reading agent is overstated. The paper says the agent "only matches" retrieval (abstract, l. 28–29; l. 468–469). On R1's own grades, the one-sentence agent's list holds 91 top-grade papers against retrieval's 84, with mean grade 3.89 vs. 3.78 (Table 7). For R2, retrieval is significantly worse than both agent lists (−0.44 and −0.45 mean grade; Table 6), and for R3 it is worse than the profile-input agent (−0.16). Evidence for R2 and R3 also comes only from an LLM judge with Spearman 0.55 against R1 (l. 1022–1024). This judge counts 22 of 176 papers as worth reading where R1 counts 151 (l. 1075–1076), and it agrees with itself only 65–74% of the time across sessions (l. 1028–1029). The conclusion that "ranking the week therefore calls no language model" is a cost argument, not a quality result, and should be stated as such.
- Scope. All three researchers are in applied mathematics, and the Experiment Check runs are cheap statistical studies (uncertainty quantification, survival analysis, adaptive testing). Whether the design works for fields where experiments need significant compute or data is untested.
Questions for authors
- Researcher assessment of ideas. Did R1 (or other researchers) assess the candidate ideas, retained and eliminated, for relevance, novelty and value? If not, can you collect such grades, ideally blind to the Desk Check verdict, and report how the retained/eliminated split looks under them?
- Grader independence. Can you re-run the §5.1 grading with case files that contain only the idea and seed abstract, without the Desk Check's premise outputs, derivations and closest-work search? How much of the 0.79-point gap remains?
- Baselines for the checker. How does the Desk Check compare with a single LLM critique call, or with a self-refine baseline, at the same elimination rate? What does the model-free replay program catch on its own?
- Relationship of R1 to the authors. Is R1 an author or otherwise involved in the system's design? Did R1 know the system's signals when grading?
- Rule-assigned grades. How do the results in §4.2 and Fig. 3 change if the 380 rule-assigned grades are excluded, or graded individually? In particular, how much of the category signal's contribution remains?
- Experiment Check correctness. For the three runs, can you provide the full specifications, registries and reviewer outputs? Did a human expert check whether the verdicts were right? How often did auditors find mismatched numbers across all runs, not just these three?
- Personalization of ideas. How much do the generated ideas depend on the profile? E.g., if the same seed papers are given with a different researcher's profile, do the ideas change, and does R1 prefer the ones made with their own profile?