- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces RW-Cite, a ten-domain benchmark for recommending related-work citations from only a manuscript’s title and abstract, together with a pipeline combining structural retrieval, SciBERT scoring, graph attention, and fusion. Its main contribution is an auditable evaluation resource that distinguishes candidate coverage from ranking and demonstrates consistent gains from combining textual and citation evidence.
Critical strengths
- Ten domain graphs and section-specific citation labels provide a substantial benchmark for recommendation from titles and abstracts.
- Graph masking, provenance records, and candidate manifests make query-citation exclusion and intermediate retrieval stages auditable.
- Fusion improves over both individual rankers in all ten domains, supporting the complementarity of textual and graph evidence.
Critical weaknesses
- Split-level masking does not establish performance when every graph edge and citation context predates its query.
- The 2.31× comparison uses different retrieval pipelines and Top-30 versus Top-400 windows, so it does not isolate ranking quality.
- The 96.2% relevance estimate comes from one LLM judging training queries, without held-out expert validation.
Questions for authors
- Can you rerun evaluation with per-query historical graph snapshots and report changes in candidate coverage and Hits@10?
- Can you compare RW-Cite and the LLM rankers on identical candidate pools and evidence, reporting quality and inference cost?
- Can domain experts blindly assess gold and non-gold recommendations on held-out queries, with agreement and confidence intervals?
2 comments
Thanks for the authors' response. I will keep my positive score. Please be sure to include the discussion into your final version (if accepted).
Thank you for your thoughtful and encouraging review. We appreciate your recognition of the ten-domain benchmark, the auditability of its evaluation protocol, and the consistent gains from combining textual and citation-graph evidence. Your questions helped us sharpen the distinction between what the benchmark measures and what the reference framework demonstrates. We are grateful for the opportunity to clarify the temporal controls, compare rankers on identical candidates, and explain the scope of the relevance assessment.
Weakness 1 and Q1. Per-query historical graphs and evidence
RW-Cite retains its lead under shared temporal controls. On the same 381 EWM queries and full-gold labels, both pipelines restricted candidate papers and citation evidence to publications before each query month. With a separate RW-Cite fit trained on pre-test data, RW-Cite led HLM–Qwen by 1.163 Hits@10 (95% paired query-bootstrap CI [0.921, 1.399]).
Holding that fit fixed, we rebuilt retrieval, citation evidence, and graph features for each query month. Relative to split masking, Hits@10 changed by −0.013 (95% paired CI [−0.110, 0.084]) and Top-400 full-gold coverage changed by −0.56 percentage points (95% paired CI [−0.94, −0.18]). Appendix A.4 reports the paired changes, and Section 5.1 reports the comparison with HLM–Qwen. The cutoff uses publication months and the available extracted text. NWFE Q2 describes the original split mask and the audit of candidate dates in its saved rankings.
Weakness 2 and Q2. Shared candidates, evidence, and inference cost
Candidate discovery. On the same 381 EWM queries, both Top-30 pools contain only papers predating the query month. The HLM-Cite pool comes from its adapted two-stage GTE retriever. With shared full-gold labels, RW-Cite achieves 21.17% Recall@30 vs. 7.97% and retrieves 3.811 more gold references per query before reranking (95% paired bootstrap CI [+3.415, +4.210]). Perfect ordering within these pools would yield at most 5.654 and 2.291 Hits@10, respectively.
Ranking on identical candidates. Qwen and the RW-Cite rankers used the same RW-Cite Top-30 IDs. Qwen and the cross-encoder received the same underlying candidate text before model-specific formatting and truncation. Fusion additionally used graph features. The original split-masked graph supplied the text and can contain contexts from same-month or later citing papers.
Qwen gains 0.194 Hits@10 over fusion on this pool (95% paired bootstrap CI [+0.058, +0.331]). It scores 1.735 Hits@10 on the HLM-Cite shortlist using the same model and one-shot example. The two Qwen runs used different candidate-text selection rules. The direct recall comparison measures the shortlist advantage independently of those rules. RW-Cite makes more gold references available at matched depth, and Qwen recovers more of those references in the Top-10 when given RW-Cite’s pool.
Inference cost and reliability. On the shared pool, Qwen used two hosted calls per query, averaging 15.82 seconds of API latency (95th percentile 18.80 seconds) and approximately 28,779 input and 1,689 completion tokens per query. In 3 of 381 queries, missing valid selections were filled in shortlist order. RW-Cite scores locally without a hosted generative API. On the same queries, original Top-400 logs record 860.91 seconds for offline cross-encoder scoring and 70.00 seconds for graph-attention scoring. These are separate batch timings on prebuilt windows.
Weakness 3 and Q3. Held-out expert relevance assessment
We agree that independent domain-expert judgments are needed to validate the relevance labels. The 96.2% figure is the share of Top-30 recommendations rated core or related by an LLM on training queries. We interpret it as an exploratory estimate of judge-assigned relevance on that sample. The primary comparisons measure recovery of observed citations with full-gold labels held fixed across methods. The supplementary assessment asks whether recommendations beyond those citations may also be relevant. Held-out expert annotation would strengthen that assessment.