- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents ResearchArcade, a multi-table graph linking ArXiv papers (decomposed into sections, paragraphs, figures, tables, citations) with OpenReview submissions, reviews, and revisions, with timestamps supporting point-in-time access. It is evaluated on six academic tasks and two evidence-discovery workflows (Citation Set Completion, Literature Set Expansion). Using degree-preserving rewired graphs and matched-pool replays, the authors show that correct citation connectivity substantially improves evidence recall, and that agent gains stem mainly from evidence access rather than sequential planning.
Critical strengths
- A sizable, integrated resource connecting paper content with peer-review and revision histories, with a continuous collection pipeline.
- Temporal cutoffs and source exclusion are carefully designed to prevent leakage.
- The rewiring and matched-replay controls are well designed, with paired bootstrap CIs and Holm correction; The finding that agents add little beyond their candidate pools is informative.
- Strong reproducibility commitments (frozen splits, prompts, trajectories, graph hashes).
Critical weaknesses
-
Table 2 offers weak support for the paper’s claims. The six tasks show that the interface runs end to end, but results are uneven, some tasks show essentially no learning (e.g., acceptance prediction is near chance), and only relative comparisons among the authors’ own models are available. The table therefore contributes little to the paper’s core argument.
-
Lack of semantic information in retrieval process. The experiments show that citation graph + BM25 substantially outperforms BM25 alone. In practice, literature retrieval routinely relies on dense retrievers trained on scientific text and on rerankers. Without such baselines, it is unclear how much gain the graph provides on top of a strong semantic retriever.
-
The value of the resource’s fine-grained structure is not established. ResearchArcade’s main distinction from prior datasets is its paragraph-level granularity and its review/revision data. In the evidence-discovery experiments, the authors attribute the gains primarily to paper-level citation connectivity (Sec. 5.3); the only fine-grained variant reported (“paragraph edges only”) performs on par with BM25 (Table 15), and figures, tables, reviews, and revisions do not appear in either workflow (Secs. 4.4.1–4.4.2). It therefore remains unclear what the evidence-discovery results demonstrate beyond the known value of citation graphs.
Questions for authors
-
In revision retrieval, reviews barely affect trained models (F1 0.260→0.265; GNN Recall@5 decreases). Are models learning positional priors?
-
Table 15: “Citation edges only” equals “Correct graph.” Is the correct graph citation-only? If so, how does this support the claim in Sec. 5.3?