- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SURF augments frozen graph-model embeddings with precomputed neighborhood signatures. Lightweight relationship-specific operators and calibrated score fusion combine structural and backbone information. All relationship-dependent computation remains on the query side, preserving an inner-product retrieval interface. Experiments evaluate academic datasets, production prediction tasks, and retrieval costs.
Critical strengths
Preserving reusable candidate representations while introducing relationship-specific structural scoring is an appealing design. The implementation requires very few task-specific parameters.
The evaluation spans multiple backbones and datasets. The paper also distinguishes exact score factorization from approximate index search and explicitly acknowledges storage and refresh costs.
Critical weaknesses
The comparisons do not establish how much benefit requires a pretrained GFM. Strong structural references—such as Bloom-signature predictors, MPLP, or calibrated heuristic fusion—are absent from the main evaluation. Table 3 also introduces calibration and learned fusion together at the stage producing the largest jump. A structural-only model and calibration-only controls are important for identifying SURF’s added value.
The balanced-negative protocol is unusual. Appendix E ranks each positive against scores from a shared pool of negative pairs, irrespective of their source node. This differs from ranking candidate destinations for a fixed query and can reward score calibration across queries. These results should be separated clearly from conventional query-wise retrieval metrics and checked using standard per-query evaluation.
The serving experiments provide limited evidence for the advertised scale. The production total exceeds one billion scored pairs, but each group contains 4,093 candidates. The latency test uses 100,000 candidates, and HNSW recall is unreported. The separate large-graph preprocessing experiment computes signatures for approximately 50,000 requested nodes. Together, these support useful components of the system, but leave full-population index memory, recall, and refresh costs unresolved.
Questions for authors
How does SURF compare with structural-only scoring and calibrated fusion of established structural predictors?
Do the balanced-negative conclusions hold under fixed-query candidate ranking?
Can you report index size and recall–latency curves on actual learned representations at larger candidate populations?
2 comments
Dear Reviewer,
Thank you again for your feedback. As the response deadline approaches, we wanted to briefly follow up on our rebuttal and the revised manuscript. Appendix G now includes the controlled calibration-versus-fusion ablation, calibrated CN/AA baselines, and structural-only comparisons (G.2–G.4). We also clarified Table 1’s query-wise retrieval protocol (G.5) and added recall–latency curves and index-size measurements on learned representations for up to 676,084 candidates (G.6, Figure 7).
We hope these additions help address the concerns you raised. Could you let us know whether any points remain unresolved or would benefit from further clarification? We would appreciate the opportunity to discuss them before the response period closes.
Thank you for your time and consideration.
Reply to Reviewer f3jo
We thank the reviewer for the constructive comments. We added component comparisons, retrieval results, and protocol clarifications in Appendix G, marked in purple.
Structural baselines. Common Neighbors (CN) and Adamic–Adar (AA) use the same RMS calibration and relation-conditioned fusion procedure as SURF, with train/validation supervision only. The additional comparisons cover ten train/test datasets, two seeds, and 4,096 filtered negatives per positive. MRR is reported as mean ± sample SD across seeds: relation tasks are weighted by original edge counts within each dataset, then datasets are averaged equally.
CN/AA improve both backbones, while SURF achieves substantially higher reported macro MRR and preserves inner-product retrieval. Tables 13–14 provide all dataset results, including cases where AA is stronger. The heuristic rows use average ties; SURF uses the saved strict-greater ranks.
Calibration versus learned fusion. The two SURF rows hold features, kernels, RMS scales, evaluation pairs, and ranking protocol fixed. Learned weights improve MRR by 5.75±1.58 and 11.80±0.22 percentage points for GraphBFF and HGT (paired seed differences). Thus, calibration alone does not explain the gains. Appendix G.2 and Table 12 report the complete controlled ablation, complementing the existing RMS-removal ablation.
Structural-only scoring. Removing pretrained embeddings yields 0.3641±0.0007 macro MRR, confirming that structure is the strongest signal in this aggregate. The backbone contributes differently across tasks: full SURF with HGT outperforms both branches on Fitness, Goodreads, and Rel-Stack. On Goodreads, HGT, structural-only scoring, and full SURF achieve 0.1329±0.0062, 0.8409±0.0018, and 0.8734±0.0089, respectively. Appendix G.4 and Tables 15–16 provide all dataset comparisons under the same ranking convention.
Retrieval evaluation. Table 1 already evaluates query-wise sampled retrieval using 10% of test positives and 10% of compatible candidates. SURF(HGT) improves MRR on 14 of 16 datasets, including all six held-out datasets. This conclusion therefore holds independently of the balanced-negative evaluation. Appendix G.5 explicitly distinguishes Table 1's candidate ranking from the shared negative-pair score pool used in the balanced-negative results.
Recall–latency curves and index footprint. Appendix G.6, Figure 7, and Table 17 add ANN measurements on learned GraphBFF + SURF representations for 100,000 candidates and all 676,084 Goodreads destinations. At structural width 256, semantic-first retrieval followed by SURF reranking gives:
Values are mean ± sample SD across two seeds after averaging five warm-query repetitions per seed. Latency combines semantic-search p95 with mean reranking time. The full-population fused index occupies 3.30 GB; the semantic index occupies 0.53 GB, plus candidate data for reranking. Figure 7 shows the recall, latency, and storage trade-offs across structural widths and both populations.
Production evaluation and scalability. We rank each positive against 4,092 sampled negatives, following the platform's internal protocol and using identical groups across models. Sampled-candidate ranking is widely used in recommender evaluation to reduce the cost of scoring the full candidate population [1]. This measures prediction quality under substantial class imbalance while making repeated production evaluation tractable. The sampled group size does not limit the population SURF can search.
SURF scores 1.105 billion pairs per backbone across three production tasks, demonstrating its ability to process a large production workload. Its MIPS formulation supports cached candidate vectors and retrieval without query-time graph traversal. In the paper's 100,000-candidate benchmark, exact SURF retrieval takes 68.5 ms; the evaluated ULTRA configuration requires an estimated 633.8 seconds. The production workload and learned-vector ANN results provide complementary evidence of scoring and retrieval scalability (Appendix G.7).
Given the additional evidence, we hope the reviewer reconsiders their recommendation.
[1] Zhao et al., [A Revisiting Study of Appropriate Offline Evaluation for Top-N Recommendation Algorithms]