acceptodds
Under review as a conference paper at ICLR 2027

RW-Cite: A Large-Scale Multi-Domain Benchmark for Citation Recommendation in Related Work Generation

Abstract

Reliable related-work generation begins with identifying the prior studies that establish a manuscript's context, differentiate its contributions, and support its claims. In the cold-start setting, however, a system must recover these references from a large and evolving literature using only the manuscript's title and abstract, without citation contexts or a predefined candidate set. We introduce RW-Cite, a multi-domain benchmark and end-to-end framework for evaluating this complete literature-discovery and citation-ranking process. RW-Cite applies multi-level topic retrieval to 3,144,903 arXiv paper metadata records and constructs domain-specific citation graphs with contextual evidence. From this corpus, we evaluate ten scientific domains spanning robotics, autonomous driving, biomedical imaging, astronomy, cosmology, condensed-matter physics, quantum computing, and scientific machine learning. The ten independently constructed graphs contain 257,393 node entries and 940,141 section-aware citation-edge entries. The RW-Cite framework combines learned structural shortlisting, a citation-aware SciBERT cross-encoder, a graph attention network (GAT), and fixed score-and-rank fusion. Across the evaluated domains, RW-Cite establishes strong citation-recommendation performance; on the EWM comparison benchmark, it attains 2.31 the Hits@10 score of the strongest evaluated frontier-LLM-based pipeline. In a preliminary analysis of 640 training queries, an LLM assessed the 30 candidates ranked highest by RW-Cite for each query, yielding 19,200 query–candidate judgments; 96.2% were judged by the LLM evaluator as core or related prior work, providing complementary evidence of recommendation relevance beyond observed citation recovery. To support transparent and reproducible research, we publicly provide the benchmark datasets, temporal splits, graph and candidate manifests, together with the construction, framework, and evaluation code. The source code is available at https://anonymous.4open.science/r/RW-Cite/; the datasets are available at https://anonymous-hf.com/a/t71z3mpp2u2i/.

Primary area:
datasets and benchmarks
Keywords:
Citation Recommendation, Scientific Literature Retrieval, Related-Work Generation, Scholarly Knowledge Graphs, Datasets and Benchmarks, AI for Science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3Z8POpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces RW-Cite, a ten-domain benchmark for recommending related-work citations from only a manuscript’s title and abstract, together with a pipeline combining structural retrieval, SciBERT scoring, graph attention, and fusion. Its main contribution is an auditable evaluation resource that distinguishes candidate coverage from ranking and demonstrates consistent gains from combining textual and citation evidence.

Critical strengths
  1. Ten domain graphs and section-specific citation labels provide a substantial benchmark for recommendation from titles and abstracts.
  2. Graph masking, provenance records, and candidate manifests make query-citation exclusion and intermediate retrieval stages auditable.
  3. Fusion improves over both individual rankers in all ten domains, supporting the complementarity of textual and graph evidence.
Critical weaknesses
  1. Split-level masking does not establish performance when every graph edge and citation context predates its query.
  2. The 2.31× comparison uses different retrieval pipelines and Top-30 versus Top-400 windows, so it does not isolate ranking quality.
  3. The 96.2% relevance estimate comes from one LLM judging training queries, without held-out expert validation.
Questions for authors
  1. Can you rerun evaluation with per-query historical graph snapshots and report changes in candidate coverage and Hits@10?
  2. Can you compare RW-Cite and the LLM rankers on identical candidate pools and evidence, reporting quality and inference cost?
  3. Can domain experts blindly assess gold and non-gold recommendations on held-out queries, with agreement and confidence intervals?
2 comments

Thank you for your thoughtful and encouraging review. We appreciate your recognition of the ten-domain benchmark, the auditability of its evaluation protocol, and the consistent gains from combining textual and citation-graph evidence. Your questions helped us sharpen the distinction between what the benchmark measures and what the reference framework demonstrates. We are grateful for the opportunity to clarify the temporal controls, compare rankers on identical candidates, and explain the scope of the relevance assessment.

Weakness 1 and Q1. Per-query historical graphs and evidence

RW-Cite retains its lead under shared temporal controls. On the same 381 EWM queries and full-gold labels, both pipelines restricted candidate papers and citation evidence to publications before each query month. With a separate RW-Cite fit trained on pre-test data, RW-Cite led HLM–Qwen by 1.163 Hits@10 (95% paired query-bootstrap CI [0.921, 1.399]).

Holding that fit fixed, we rebuilt retrieval, citation evidence, and graph features for each query month. Relative to split masking, Hits@10 changed by −0.013 (95% paired CI [−0.110, 0.084]) and Top-400 full-gold coverage changed by −0.56 percentage points (95% paired CI [−0.94, −0.18]). Appendix A.4 reports the paired changes, and Section 5.1 reports the comparison with HLM–Qwen. The cutoff uses publication months and the available extracted text. NWFE Q2 describes the original split mask and the audit of candidate dates in its saved rankings.

Weakness 2 and Q2. Shared candidates, evidence, and inference cost

Candidate discovery. On the same 381 EWM queries, both Top-30 pools contain only papers predating the query month. The HLM-Cite pool comes from its adapted two-stage GTE retriever. With shared full-gold labels, RW-Cite achieves 21.17% Recall@30 vs. 7.97% and retrieves 3.811 more gold references per query before reranking (95% paired bootstrap CI [+3.415, +4.210]). Perfect ordering within these pools would yield at most 5.654 and 2.291 Hits@10, respectively.

Ranking on identical candidates. Qwen and the RW-Cite rankers used the same RW-Cite Top-30 IDs. Qwen and the cross-encoder received the same underlying candidate text before model-specific formatting and truncation. Fusion additionally used graph features. The original split-masked graph supplied the text and can contain contexts from same-month or later citing papers.

RankerHits@10
Structural order3.265
Cross-encoder3.709
Graph attention3.488
Score-and-rank fusion3.761
Qwen Analyzer–Decider3.955

Qwen gains 0.194 Hits@10 over fusion on this pool (95% paired bootstrap CI [+0.058, +0.331]). It scores 1.735 Hits@10 on the HLM-Cite shortlist using the same model and one-shot example. The two Qwen runs used different candidate-text selection rules. The direct recall comparison measures the shortlist advantage independently of those rules. RW-Cite makes more gold references available at matched depth, and Qwen recovers more of those references in the Top-10 when given RW-Cite’s pool.

Inference cost and reliability. On the shared pool, Qwen used two hosted calls per query, averaging 15.82 seconds of API latency (95th percentile 18.80 seconds) and approximately 28,779 input and 1,689 completion tokens per query. In 3 of 381 queries, missing valid selections were filled in shortlist order. RW-Cite scores locally without a hosted generative API. On the same queries, original Top-400 logs record 860.91 seconds for offline cross-encoder scoring and 70.00 seconds for graph-attention scoring. These are separate batch timings on prebuilt windows.

Weakness 3 and Q3. Held-out expert relevance assessment

We agree that independent domain-expert judgments are needed to validate the relevance labels. The 96.2% figure is the share of Top-30 recommendations rated core or related by an LLM on training queries. We interpret it as an exploratory estimate of judge-assigned relevance on that sample. The primary comparisons measure recovery of observed citations with full-gold labels held fixed across methods. The supplementary assessment asks whether recommendations beyond those citations may also be relevant. Held-out expert annotation would strengthen that assessment.

Thanks for the authors' response. I will keep my positive score. Please be sure to include the discussion into your final version (if accepted).

Reviewer_wSs8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on cold-start citation recommendation: given only a manuscript's title and abstract, and no supplied candidate pool, the system must find and rank ten prior papers worth citing.

The contribution is two-fold. First, RW-Cite, a benchmark of ten domain-specific citation graphs built automatically from 3.1M arXiv records, with temporal splits and leakage masking, plus an auditable evaluation contract that keeps candidate coverage and ranking quality separate. Second, a reference framework that chains together retrieval, a learned structural shortlist, a citation-aware cross-encoder, a graph attention network, and score fusion.

On the experiments side, fusion beats every single channel in all ten domains, and on EWM the framework reaches a Hits@10 of 2.31x the strongest LLM pipeline evaluated. Datasets and code are public.

Critical strengths

The evaluation contract is the most substantive part of this paper. Pulling candidate coverage apart from ranking quality, and requiring every system to run end-to-end with its own candidate generation, is a protocol design that reranking-centric benchmarks have not had.

It also yields a genuinely informative diagnosis: the strongest external baseline fails at candidate discovery, not at ranking -- which directly supports the paper's central claim that retrieval has to be part of the evaluated task.

The contract is also operational: data, manifests, per-query outputs, and code are all out there; and the analysis of non-gold recommendations is done with a care you don't often see in benchmark papers.

Critical weaknesses

Limited novelty -- largely an assembly of existing techniques. There is no new model and no new module in the framework, and the benchmark is built from public arXiv data with routine procedures that others could replicate without much trouble. What's left is careful integration and engineering rigor, and I'm not convinced that alone clears the contribution bar.

The end-to-end comparison is not symmetric. The LLM baselines can only pick from the retriever's top-30 candidates and are not allowed to search further, while RW-Cite ranks over a window of several hundred candidates produced by its own multi-stage construction -- the two sides are working with very different amounts of recoverable candidates. On top of that, RW-Cite's training cost is substantially higher with no compute-matched comparison, and the external baselines were only run on EWM. So the 2.31x headline rests on a single domain under an unequal setup, and it's hard to say how much of it is the design and how much is the setup itself.

Questions for authors
  1. External baselines were compared on EWM only (Table 2). Could you run the same end-to-end comparison on two or three additional domains? That would tell us whether the single-domain result is representative at all.

  2. How sensitive are the results to the fixed fusion weights? For the structural shortlister, which individual features matter most when removed?

  3. The quality of Figure 1 need improvement, the letter spacing is inconsistent throughout.

2 comments

Thank you for highlighting the value of measuring candidate coverage and ranking separately. We expanded the comparisons across domains, tested fusion and structural features, and redrew Figure 1.

Weakness 1. What is new in RW-Cite?

RW-Cite contributes a benchmark and an evidence-aware reference framework for cold-start related-work recommendation. Its ten automatically constructed graphs link section-aware citation labels, candidate-side evidence, explicit graph masks, and intermediate candidate manifests. Together, these artifacts show whether a reference was absent from the corpus, lost during shortlisting, or ranked too low.

The new controls show why that distinction matters. On EWM, RW-Cite and the adapted HLM-Cite retriever recover 6.115 vs. 2.304 gold references per query at depth 30 (paired gain +3.811, 95% bootstrap CI [+3.415, +4.210]). On the same RW-Cite Top-30, Qwen reaches 3.955 Hits@10 and fusion reaches 3.761. A strong ranker can still be limited by candidate discovery, which the benchmark measures directly.

The reference framework combines citation evidence throughout retrieval and ranking. Learned structural shortlisting brings graph-supported papers into a tractable window, and fusion combines a citation-aware cross-encoder with graph attention. With the original Top-400 candidates fixed, fusion improves EWM Hits@10 over structural order from 3.265 to 4.000 (paired gain +0.735, 95% bootstrap CI [+0.577, +0.890]). Fusion improves on structural order and on each individual scoring channel in all ten domains by point estimate. These controls quantify the ranking contribution with candidate availability held fixed.

Weakness 2 and Q1. Candidate-window fairness, compute, and additional domains

The reported 2.31× compares the complete EWM pipelines. At a matched depth of 30, we evaluated both shortlists on the same 381 queries and full-gold labels. RW-Cite’s structural ordering alone reaches 3.265 Hits@10, above the HLM shortlist’s oracle ceiling of 2.291. This shows a candidate-discovery advantage at matched depth before neural reranking. Our response to 3Z8P Q2 reports ranking on identical candidates and the scope of the inference measurements.

We extended the comparison to the original SQC and GW test splits, with shared full-gold labels and each pipeline’s native candidates. RW-Cite leads HLM–Qwen on Hits@10 in all three evaluated domains.

DomainRW-Cite Hits@10HLM–Qwen Hits@10Paired gain (95% bootstrap CI)
SQC2.7272.239+0.488 [0.239, 0.730]
GW3.6582.698+0.961 [0.669, 1.253]

RW-Cite also leads on nDCG@10 and full-gold Recall@10 in both additional domains. HLM–Qwen has slightly higher MRR@10 on SQC and GW, while RW-Cite leads on EWM. In the separate ten-domain dense and graph comparison, RW-Cite leads Hits@10 in eight domains and has the highest mean (3.081 vs. 2.690 for citation voting).

Q2. Fusion weights and structural features

With EWM Top-400 candidates and CE/GAT scores fixed, we varied both the GAT score weight and rank-fusion weight over 0.3, 0.4, 0.5, and the reciprocal-rank offset over 10, 20, 30. Across 27 settings, Hits@10 ranged from 3.898 to 4.000 and Hits@30 from 7.294 to 7.496. All settings were within 0.102 Hits@10 (about 2.6%) of the reported value. Score-only fusion reached 3.924 Hits@10, and adding rank evidence at the reported setting gained 0.076 (95% paired bootstrap CI [+0.013, +0.139]).

We refitted the shortlister after dropping each feature, using the original 2,910 fitting and 513 development queries. The full model reproduced 62.81% development Recall@400, conditional on gold being present in the cached universe. Removing lexical similarity caused the largest drop (2.51 percentage points), followed by log out-degree (1.19), log support (1.16), and embedding similarity (1.15). These are development-set shortlisting results. NWFE Q4 reports the fixed-window test of reranking.

Q3. Figure 1 typography

We redrew Figure 1 as an SVG, standardized label typography and spacing, and checked legibility at manuscript size.

Thank you for the detailed response and additional experiments.

The dataset is valuable and additional statistics prove better performance of the proposed framework in some settings. However, the novelty of the dataset and method is still expected to be more clearly.

Reviewer_NWFEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RW-Cite tackles cold-start citation recommendation: given only a paper’s title and abstract, find the references it should cite in its Introduction and Related Work. It contributes an automatically built benchmark of domain-specific arXiv citation graphs, spanning fields from robotics to astrophysics and pathology, with section-aware citation labels and time-based splits that hide test papers’ citations. It also offers a reference pipeline that expands candidates through the citation graph and reranks them with a SciBERT cross-encoder and a graph attention network. For validation, the authors compare against keyword, dense-retrieval and LLM-reranking baselines on a single domain, run the same pipeline separately in every domain, and ablate each reranker and the fusion rule. They also use an LLM judge to check whether recommendations beyond the actual citations are still relevant, and run a small case study generating citation sentences for the selected papers.

Critical strengths
  1. The task and evaluation design are realistic and unusually careful. Systems must find references from only a title and abstract instead of reranking a supplied pool, and saved intermediate candidate lists show whether a missed paper was never retrieved or just ranked badly. Candidates are described by how earlier papers cited them while the query's own citing sentences stay hidden. The buffer month, target-edge removal during GAT training, and exclusion of all known positives from the negatives address leakage paths that often inflate citation-recommendation results. This is consequential because it mirrors how research agents actually meet the problem, and it is the main reason I see the benchmark as worth fixing rather than discarding.

  2. The benchmark brings citation recommendation to fields it rarely reaches. Most of its domains are in physics and astronomy, and all of them come from a single automated pipeline that can be pointed at new fields. This matters because evaluation resources outside CS and biomedicine are scarce, so once the validity concerns are addressed, the suite could become a genuinely useful shared resource.

Critical weaknesses
  1. The experiments cannot attribute the reported gains to the proposed method. External baselines are text-only retrievers run on a single domain, the LLM pipelines are capped by a weak retriever's top-30, simple graph-aware baselines such as nearest-neighbor reference voting and a citation-count prior are absent, and the ablation omits a structural-only ranker. This is decisive because the paper's main empirical claims, strong performance across domains and a large margin over LLM pipelines, rest on these comparisons, and as designed they cannot separate a modeling advance from extra graph information or a classic collaborative-filtering heuristic.

  2. The validity of the benchmark, the paper's central contribution, is not established. Cited-paper closure uses test papers' bibliographies to build a small closed-world corpus guaranteed to contain every gold reference, the extraction and citation-resolution pipeline is never validated, and the promised coverage-versus-ranking diagnostics are not reported. This matters because a benchmark paper is judged on whether others can trust and reuse its labels and difficulty, and here neither the label quality nor the realism of the search space is demonstrated.

Questions for authors
  1. Hits@30 divided by R@30 is below ten for WSI and FNO, which should be impossible if every query has at least ten gold references; do some queries have fewer, what is the gold-outdegree range, and what are the per-domain gold-set size distributions?
  2. Are outgoing edges of non-admitted seed papers published in the frontier or test window removed, and does RW-Cite exclude candidates published after each test query as the baselines do?
  3. What fraction of test gold papers have no visible incoming citations after masking, and how do results change if cited-paper closure uses only training-period papers?
  4. What is the gold coverage of the candidate universe and the Top-400 window, and how well does the structural shortlister rank on its own?
  5. How do BGE and SciBERT-NTX (with and without candidates expanded by the same citation sentences), nearest-neighbor reference voting and a citation-count prior perform across all ten domains?
  6. What fraction of bibliography entries resolve in each domain, given that admission rates range from about three quarters of seeds to under a quarter, and how accurate are section detection and citation resolution on a manually checked sample?
1 comment

Thank you for identifying the recall inconsistency and asking how candidate discovery, graph construction, and reranking affect the results.

Q1. Gold-set sizes and recall. Every test query has at least 11 gold references. The original RW-Cite recall denominator included only gold references inside the Top-400. We recomputed recall and nDCG against the full frozen gold sets. In whole-slide pathology and neural-operator turbulence, corrected Recall@30 is 26.29% and 24.55%, respectively, and Hits@30 remains 4.876 and 4.667. Both domains have 11–52 gold references per query. The revised appendix reports all ten gold-set distributions.

Q2. Temporal masking and candidate eligibility. The original mask removed outgoing edges only from sources in the test or frontier manifests. Non-admitted seeds could retain edges even in those months. RW-Cite could retrieve same-month or later papers, whereas external baselines required earlier candidates. We clarified both rules. EWM Top-400 windows contained 64 later-month occurrences across 45 queries. The final Top-10 contained zero later-month and two same-month candidates. Filtering these rankings and backfilling with scores fixed changed Hits@10 from 4.000 to 4.005. See 3Z8P Q1 for the per-query experiment that rebuilds retrieval and scoring.

Q3. Incoming edges and corpus closure. In the masked EWM graph, 23.09% of distinct gold papers had no incoming edge, accounting for 10.72% of gold pairs. The pair-level share ranged from 13.59% to 30.99% in the other nine domains. Keeping all independently retrieved seeds and allowing only seeds published before the first EWM test month to add papers through citation closure retained 25,322 of 27,705 papers, with 89.56% mean full-gold corpus coverage across 381 queries. Restricting the saved rankings to that corpus, with retrieval and scores fixed, left Hits@10 at 4.000 (paired change 0.000, 95% bootstrap CI [−0.011, +0.011]).

Q4. Coverage and reranking. The EWM candidate universe and structural Top-400 covered 82.41% and 52.67% of full gold, respectively. We verified query IDs, gold counts, and universe sizes against the split and scoring graph. With Top-400 candidates fixed, fusion improved Hits@10 over structural order from 3.265 to 4.000 (paired gain +0.735, 95% bootstrap CI [+0.577, +0.890]). Fusion had a higher point estimate in all ten domains.

Q5. Cross-domain baselines. We evaluated 2,538 queries from the ten original splits against the same full-gold labels. External controls restrict candidates and citation evidence to earlier months. RW-Cite retains its split mask and native retrieval. The table averages results across domains.

MethodMean Hits@10Mean full-gold Recall@30
RW-Cite3.08122.30%
Citation voting2.69018.44%
BGE + citation sentences2.34516.76%
BGE2.26116.05%
SciBERT–NTX1.98914.69%
SciBERT–NTX + citation sentences1.97814.93%
Citation-count prior1.1398.21%

RW-Cite leads in eight domains and improves mean Hits@10 by 14.5% over citation voting. Pointwise paired 95% bootstrap intervals favor RW-Cite over the best added baseline in five domains. BGE with citation sentences leads by 0.082 in exoplanet astronomy and 0.039 in strongly correlated electrons, with both intervals spanning zero. Appendix D.4 gives per-domain results.

Q6. Citation extraction and resolution. In the EWM reparse, 215,849 of 268,078 citation occurrences (80.52%) mapped to canonical IDs before repeat-target and self-citation filtering, among papers with loadable bibliographies. In the other nine domains, all 64,955 gold pairs across 2,157 test queries have a stored sentence and a target in the scoring graph. All 745,056 stored scoring edges also have nonempty citation text. These checks cover retained links.

In a 12-paper EWM pilot, the extractor found all 12 Introductions and nine conventionally headed prior-work sections. It missed two nonstandard background passages and extracted some keys from source comments. Of 48 sampled keys, 47 appeared in the extracted spans. All 36 resolved targets agreed with the cited title or ID, and 33 links appear in the frozen graph. We describe the fallback rule and error modes in the revision.

Weaknesses 1–2. Attribution, fairness, and validity. The fixed-window control quantifies fusion’s gain (Q4), and the ten-domain comparisons show how performance varies by field (Q5). On EWM, RW-Cite’s shortlist recovers 3.811 more gold references per query at depth 30 than adapted HLM-Cite, while Qwen leads fusion on RW-Cite’s shared Top-30 (3.955 vs. 3.761 Hits@10, 3Z8P Q2). The revision distinguishes split masking from per-query cutoffs and documents the coverage and extraction checks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.