- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SEEDER is a first-stage KG retriever. It seeds nodes by entity matching and similarity, expands them into a candidate subgraph of at most 500 nodes, and encodes it with a query-conditioned sparse Graph Transformer; a REINFORCE-trained policy head selects up to 20 answers and a BPR head ranks them. A theory section separates query-independent dense retrieval from query-conditioned message passing. On three STaRK graphs with three encoders, SEEDER beats same-encoder dense and k-hop baselines.
Critical strengths
- The seed-expand-select design fits multi-hop KG retrieval.
- Gains are consistent across all graphs and encoders, with mean and std over five seeds in Table 1.
Critical weaknesses
- The released configs do not match the main table. prime_ada.yaml uses PCA 256, VN only and three hops, and prime_qwen.yaml enables the virtual node the paper turns off.
- Sparse feedback is a central motivation, but every experiment trains on the full answer sets losses() in seeder/training.py, and the feedback ratio is never varied.
Questions for authors
- Could you release configs for all nine rows? Was the validation split used for both hop-quota search and checkpoint selection?
- Could you simulate partial feedback on fixed candidate graphs?
4 comments
the feedback ratio is never varied.
We did another study to address the varying feedback ratio part of this comment. For each 20-node proposal during the training process, we randomly get labels for only 2, 5, 10, or 20 nodes. The reward is correct judged nodes / number of judged nodes, so it does not require the full answer count. The auxiliary ranking loss uses only judged nodes.
| Feedback per proposal | Hit@1 (%) | Hit@5 (%) | MRR@20 (%) | Recall@20 (%) |
|---|---|---|---|---|
| 10% (2/20) | 32.99 ± 1.05 | 62.43 ± 1.53 | 45.80 ± 1.23 | 70.59 ± 1.85 |
| 25% (5/20) | 41.18 ± 1.54 | 69.75 ± 0.78 | 53.73 ± 1.25 | 76.76 ± 0.51 |
| 50% (10/20) | 45.16 ± 0.90 | 73.22 ± 0.88 | 57.45 ± 0.67 | 78.62 ± 0.40 |
| 100% (20/20) | 46.15 ± 0.80 | 73.58 ± 0.84 | 58.10 ± 0.66 | 78.72 ± 0.50 |
Using feedback on half of each selected set gives almost the same Recall@20 as full feedback (78.62% vs. 78.72%). With feedback on only one-quarter, Recall@20 is still 76.76%. These results show that SeedER can learn well from partial feedback.
Thanks for the follow-up. Checking the saved configs and adding the partial-feedback experiment (with 2, 5, 10, and 20 judged nodes) answers my two main questions.
For the revised version, please include all yaml configs in the code release. Also, please clarify whether those judged nodes are picked fresh for each proposal during training or fixed once per query.
I will keep my positive score.
Thank you for reviewing our work and rebuttal and for your thoughtful feedback. We are glad our results addressed your questions. We will provide clean code and all finalized YAML configs with the revised paper.
Judged nodes are sampled randomly afresh for each proposal on every training visit. Our RL update learns from reward differences across the eight proposals for each query. Over the full 15 epochs (120 proposals per query), the median numbers of unique judged nodes per query were 72, 107, 136, and 163 for the 2/20, 5/20, 10/20, and 20/20 settings, respectively. We will clarify this in the revision.
Thank you for taking the time to carefully review our work and code. Below, we address each of your points.
Thank you for catching this mistake in the submitted code. We checked the saved configurations and test results for all five seeds of both STaRK-PRIME runs. Appendix Tables 7 and 8 accurately describe the configurations that produced Table 1; the two YAML files in the supplement contain earlier settings, and were incorrectly labeled as the main-table configurations.
The submitted code supports both configurations through the YAML files; we will include the exact configurations used for each Table 1 result in the public release.
We agree that the recall as used in the submission requires knowing the number of correct answers. We thus conducted experiments that instead use the number of correct answers divided by the total number of returned nodes (20), which is the precision, and does not require this knowledge. The performance is similar; we will report these results in the revised paper.
Yes, we will release them. The ones in the paper are accurate; we will update the code too. And yes, we have used the validation set for this purpose.
Yes, see above.