acceptodds
Withdrawn from ICLR 2027

ResearchArcade: Graph Interface for Automated Research

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

AutoResearch systems increasingly assist with literature discovery, scientific writing, peer review, and manuscript revision. Yet they commonly rely on fragmented, task-specific views of academic data that fail to expose relations across research artifacts. We introduce ResearchArcade, a graph interface that unifies multi-source, multi-modal, and temporally evolving academic data for diverse research models and workflows. ResearchArcade organizes ArXiv papers and OpenReview peer-review activities in a coherent multi-table graph and exposes task-relevant neighborhoods through a common interface. We evaluate ResearchArcade across six academic tasks spanning the research process and two evidence-discovery workflows under both retrieval and agentic settings. The task-specific evaluations show that ResearchArcade supports diverse predictive and generative models and enables tasks that combine cross-source and multi-modal information. Controlled evaluations in the two AutoResearch workflows further show that correct graph connectivity enhances evidence-discovery capabilities in both fixed retrieval pipelines and interactive agents. Together, these findings establish ResearchArcade as a general graph interface for supporting diverse research models and workflows. Code, frozen splits, and evaluation artifacts are provided in the anonymous supplementary repository.

Primary area:
datasets and benchmarks
Keywords:
AutoResearch, Academic Knowledge Graphs, Scientific Literature Retrieval, Research Agents, Multimodal Academic Data, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_k4yxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ResearchArcade, a multi-table graph linking ArXiv papers (decomposed into sections, paragraphs, figures, tables, citations) with OpenReview submissions, reviews, and revisions, with timestamps supporting point-in-time access. It is evaluated on six academic tasks and two evidence-discovery workflows (Citation Set Completion, Literature Set Expansion). Using degree-preserving rewired graphs and matched-pool replays, the authors show that correct citation connectivity substantially improves evidence recall, and that agent gains stem mainly from evidence access rather than sequential planning.

Critical strengths
  • A sizable, integrated resource connecting paper content with peer-review and revision histories, with a continuous collection pipeline.
  • Temporal cutoffs and source exclusion are carefully designed to prevent leakage.
  • The rewiring and matched-replay controls are well designed, with paired bootstrap CIs and Holm correction; The finding that agents add little beyond their candidate pools is informative.
  • Strong reproducibility commitments (frozen splits, prompts, trajectories, graph hashes).
Critical weaknesses
  1. Table 2 offers weak support for the paper’s claims. The six tasks show that the interface runs end to end, but results are uneven, some tasks show essentially no learning (e.g., acceptance prediction is near chance), and only relative comparisons among the authors’ own models are available. The table therefore contributes little to the paper’s core argument.

  2. Lack of semantic information in retrieval process. The experiments show that citation graph + BM25 substantially outperforms BM25 alone. In practice, literature retrieval routinely relies on dense retrievers trained on scientific text and on rerankers. Without such baselines, it is unclear how much gain the graph provides on top of a strong semantic retriever.

  3. The value of the resource’s fine-grained structure is not established. ResearchArcade’s main distinction from prior datasets is its paragraph-level granularity and its review/revision data. In the evidence-discovery experiments, the authors attribute the gains primarily to paper-level citation connectivity (Sec. 5.3); the only fine-grained variant reported (“paragraph edges only”) performs on par with BM25 (Table 15), and figures, tables, reviews, and revisions do not appear in either workflow (Secs. 4.4.1–4.4.2). It therefore remains unclear what the evidence-discovery results demonstrate beyond the known value of citation graphs.

Questions for authors
  1. In revision retrieval, reviews barely affect trained models (F1 0.260→0.265; GNN Recall@5 decreases). Are models learning positional priors?

  2. Table 15: “Citation edges only” equals “Correct graph.” Is the correct graph citation-only? If so, how does this support the claim in Sec. 5.3?

Reviewer_8K7tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RESEARCHARCADE, a unified graph interface connecting academic data from arXiv and OpenReview, including paper text, figures, tables, citations, reviews, and manuscript revisions. It provides a common interface for retrieving task-relevant neighborhoods to support different research models and workflows. The authors evaluate the framework on six academic tasks and two evidence-discovery workflows. Comparisons with degree-preserving rewired graphs indicate that correct citation connectivity improves evidence discovery in both fixed retrieval pipelines and interactive agents.

Critical strengths
  1. The code is publicly available, which makes it easier to reproduce the experiments and use the framework.
  2. The paper tests the framework on six academic tasks, providing evidence that it is useful beyond a single application.
  3. The ablation studies help interpret the results by showing how graph structure, figures, tables, and reviewer feedback affect performance on the evaluated tasks.
Critical weaknesses
  1. The paper’s novelty needs to be clearer. Existing systems already use graphs to organize academic literature and support research tasks. What does RESEARCHARCADE enable that these systems cannot do, beyond bringing more data sources into a common interface?
  2. The evaluation of generated text does not directly establish its scientific correctness. Similarity scores and LLM judgments may favor a well-written revision or rebuttal even if it contains unsupported claims or does not address the reviewer’s concern. The LLM judges should be calibrated against expert assessments, and a human evaluation is needed to check factual accuracy, evidence support, and whether the responses address the feedback.
  3. The quality of the graph itself needs more attention. Errors in extracting content, matching papers across platforms, or generating annotations could affect several downstream tasks. A manual assessment of these errors would make the results more convincing.
  4. Although the evaluation covers six tasks, it mainly uses computer science papers and ICLR review data. This supports the framework’s usefulness across tasks, but its applicability to other scientific fields remains untested
Questions for authors
  1. What filtering or quality-control criteria are used to select arXiv papers for inclusion, and how do the authors assess whether low-quality or unsupported claims affect downstream results?

  2. What is the motivation? As I understand it, the paper introduces a unified graph interface that connects academic artifacts across sources, modalities, and manuscript versions. However, existing systems such as SGHA (arXiv:2608.17501) already structure literature into evidence-linked graphs for research-problem discovery. Could the authors clarify which limitations of existing graph-based systems RESEARCHARCADE addresses, beyond providing a broader dataset and a common interface, and why those limitations matter for the evaluated workflows?

Note: A convincing response clarifying the paper’s motivation and contribution over existing work could raise my rating to weak rejection. If the authors also adequately address the remaining concerns about data quality and evaluation as well, I would consider weak acceptance.

Reviewer_UhjqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ResearchArcade, a unified graph-based interface for representing heterogeneous, multimodal, and temporally evolving academic data. The authors argue that existing systems supporting academic workflows, such as literature discovery, scientific writing, peer review, and manuscript revision, typically rely on fragmented and task-specific representations that fail to capture relationships across different research artifacts. To address this limitation, ResearchArcade organizes information from arXiv and OpenReview into a unified multi-table graph that connects papers, citations, figures, tables, reviews, and manuscript revisions. The proposed interface enables models and agents to access task-relevant neighborhoods of this graph through a common representation. The authors evaluate ResearchArcade on six academic tasks, along with two evidence-discovery workflows under retrieval-based and agentic settings. The results demonstrate the applicability of the proposed interface across different predictive and generative tasks. Additionally, experiments using degree-preserving graph rewiring suggest that preserving citation connectivity improves evidence-discovery performance, highlighting the potential value of relational information in academic workflows.

Critical strengths
  1. S1. Natural representation of heterogeneous academic data. The paper adopts a graph-based formulation that is well suited to capturing the heterogeneous and relational nature of academic information. Representing papers, citations, reviews, revisions, and other research artifacts within a shared relational structure is an intuitive and potentially useful design choice.
  2. S2. Broad experimental evaluation. The authors evaluate the proposed framework across multiple tasks and experimental settings, including predictive, generative, and agentic workflows. This diversity of experiments is a strength, as it illustrates the potential applicability of the interface to different academic use cases.
Critical weaknesses
  1. W1. Unclear positioning within the AutoResearch literature. One of my main concerns is the use of the term "AutoResearch," which is broad and can refer to substantially different research directions. In particular, AutoResearch is often associated with autonomous scientific discovery, AI co-scientists, and agent-based systems that formulate hypotheses, design experiments, and iteratively conduct research. In contrast, the contribution of this paper appears primarily focused on structuring academic data and supporting downstream literature- and peer-review-related tasks. While these are valuable research problems, the connection to the broader vision of autonomous scientific research is not sufficiently established. I encourage the authors to explicitly define what they mean by AutoResearch, ideally in the abstract and introduction, and clarify which aspects of the research process their framework aims to support. Otherwise, the current positioning risks overstating the scope of the contribution.
  2. W2. Presentation and narrative coherence. The paper would benefit from substantial improvements in writing and organization, particularly in the introduction. Several passages feel generic or repetitive, making it difficult to identify the precise research gap, the technical challenges being addressed, and the central methodological contribution. The introduction would be stronger if it followed a more coherent progression from the limitations of existing academic data representations to the motivation for the proposed interface and its specific advantages. Relatedly, the contributions are not presented with sufficient clarity. I recommend explicitly separating the data representation, the proposed interface, and the experimental evaluation, and explaining what is technically novel about each component.
  3. W3. Insufficient explanation of the temporal and dynamic aspects. The paper repeatedly emphasizes that ResearchArcade represents "temporally evolving" or "dynamically evolving" academic data. However, it is not sufficiently clear what these terms mean in the context of the proposed framework. For example, does the graph explicitly model temporal changes in its structure, such as new citations, review updates, or manuscript revisions? Does the framework support time-dependent queries or incremental updates? Or does "dynamic" simply refer to the fact that academic data evolves over time? This distinction is important because representing multiple versions of research artifacts in a static graph is different from introducing a framework that explicitly models graph evolution. The authors should provide a more precise definition of the temporal setting and clarify which dynamic capabilities are actually supported and evaluated.
  4. W4. Lack of a clearly articulated problem formulation. Although the paper introduces a unified graph representation and evaluates it across several tasks, I found it difficult to identify a precise and self-contained problem definition. In particular, it is unclear whether the primary research problem concerns heterogeneous data integration, graph representation learning, relational retrieval, or the development of infrastructure for research agents. I encourage the authors to include a dedicated problem formulation that specifies the input data, the graph representation, the supported operations, and the intended outputs. This would help readers distinguish the underlying research problem from the particular implementation and downstream applications.
  5. W5. Insufficient motivation for the unified interface and limited comparison with related approaches. Another concern is that the necessity of introducing a unified graph layer is not convincingly established. While organizing heterogeneous research artifacts into a graph is a reasonable design choice, the paper does not sufficiently explain why this particular interface is needed or what limitations of existing relational and graph-based approaches it addresses. In particular, the proposed framework appears closely related to relational deep learning, where heterogeneous information distributed across multiple interconnected tables can be modeled directly. A comparison with representative relational deep learning approaches would help clarify the novelty and practical advantages of ResearchArcade. The current experiments provide evidence that citation connectivity matters, but this does not necessarily establish that the proposed unified interface is superior to alternative relational representations or existing graph-based data integration frameworks. Stronger baselines or targeted ablations would help support this claim.
Questions for authors
  1. Q1. Relationship to relational deep learning. The proposed framework seems closely related to relational deep learning, particularly approaches that learn from heterogeneous, interconnected tables. Could the authors clarify how ResearchArcade differs from existing relational deep learning frameworks, including those discussed in https://arxiv.org/pdf/2506.16654? It would be valuable to include comparisons with representative relational learning baselines, where applicable, to better establish the novelty and advantages of the proposed approach.

  2. Q2. Necessity of the unified graph interface. Could the authors elaborate on why a unified graph interface is necessary for the considered tasks? In particular, what specific capabilities does ResearchArcade provide that cannot be achieved through existing relational databases, heterogeneous graph representations, or task-specific retrieval pipelines? An ablation comparing the unified representation against alternative data organization strategies would help substantiate this design choice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.