- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the computational redundancy introduced by fixed-size candidate sets in conventional entity disambiguation (ED) systems and proposes a lightweight adaptive candidate pruning method that requires no additional model training. Inspired by conformal prediction, the proposed method defines a nonconformity score as the difference between each candidate's generator score and that of the top-ranked candidate. A threshold is selected using the development set to adaptively determine the candidate set size for mentions with different levels of ambiguity, thereby reducing the computational cost of cross-encoder reranking. The authors conduct experiments on benchmarks including ZELDA and GERBIL, evaluating the method with different candidate generators, independently trained ED pipelines, and initial candidate set sizes. In the main ZELDA experiments, the method reduces the average number of candidates passed to the reranker from 30 to 1.79, achieves a 4.56× inference speedup, and improves macro-average accuracy from 82.7% to 83.5%.
Critical strengths
S1. The method performs adaptive pruning using existing candidate generator scores without requiring additional model training, substantially reducing cross-encoder inference costs while maintaining or improving prediction accuracy.
S2. The paper compares against fixed-size and adaptive pruning baselines, conducts additional experiments across different encoders, candidate set sizes, and benchmarks, and uses Rescue–Harm analysis to explain the accuracy improvements resulting from pruning.
Critical weaknesses
W1. Figure 3 on page 7 refers to the average candidate set size, whereas the experimental setup on pages 5–6 specifies that mentions retaining only one candidate are counted as requiring zero reranked candidates. The authors should clarify the distinction between the number of retained candidates and the number of candidates actually passed to the reranker.
W2. Although improving inference efficiency is a primary objective, Equation (11) selects the pruning threshold mainly based on development-set accuracy, considering candidate count only when multiple thresholds achieve the same accuracy. Despite the existing threshold sensitivity analysis, further evaluation under explicit computational budget constraints would help establish whether the strategy achieves a favorable accuracy–efficiency trade-off.
Questions for authors
Q1. Does the "Average Candidate Set Size" reported in Figure 3 actually refer to the number of candidates passed to the cross-encoder? Could the authors clarify the distinction between retained candidate set size and the number of candidates requiring reranking, and ensure consistent terminology across the figures and tables?
Q2. Table 13 in Appendix D.1 shows that 74.3% of mentions retain only one candidate after pruning, allowing them to bypass cross-encoder inference. Could the authors further quantify the respective contributions of bypassing reranking entirely and reducing the number of reranked candidates for mentions with multiple retained candidates to the overall inference speedup?
Q3. The proposed method relies on score differences relative to the top-ranked candidate without explicitly accounting for the reliability of that candidate. Could the authors analyze the behavior of the pruning strategy when the top-ranked candidate is incorrect or the candidate generator exhibits low confidence, particularly regarding the resulting pruning errors?
4 comments
Official Comment by Authors (2/2)
Q1. Does the "Average Candidate Set Size" reported in Figure 3 actually refer to the number of candidates passed to the cross-encoder? Could the authors clarify the distinction between retained candidate set size and the number of candidates requiring reranking, and ensure consistent terminology across the figures and tables?
Thank you for this question. We clarified this point in our response to your W1 above. We kindly ask you to refer to it.
Q2. Table 13 in Appendix D.1 shows that 74.3% of mentions retain only one candidate after pruning, allowing them to bypass cross-encoder inference. Could the authors further quantify the respective contributions of bypassing reranking entirely and reducing the number of reranked candidates for mentions with multiple retained candidates to the overall inference speedup?
Thank you for this question. We have conducted additional analysis to separately examine the efficiency gains of bypassing reranking entirely and reducing the number of reranked candidates for mentions with multiple retained candidates. The results show that the former accounts for 67.5% of the inference-time reduction our method achieves relative to No-Pruning, while the latter accounts for 32.5%. These results indicate that bypassing reranking is a larger source of inference-time savings, while pruning candidate sets for mentions that still require reranking provides a substantial additional efficiency gain.
We have added the details of this analysis to Appendix D.3 of our revised manuscript.
Q3. The proposed method relies on score differences relative to the top-ranked candidate without explicitly accounting for the reliability of that candidate. Could the authors analyze the behavior of the pruning strategy when the top-ranked candidate is incorrect or the candidate generator exhibits low confidence, particularly regarding the resulting pruning errors?
Thank you for this question. We added an analysis of pruning behavior by top-ranked candidate correctness and candidate generator confidence, using the same dual-encoder pipeline as the main experiment. For correctness, we group mentions according to whether the top-ranked candidate is correct and whether gold is in the initial candidate set. For confidence, we group all test mentions into low-, medium-, and high-confidence categories using tertile boundaries of the top-two score margin. We show the results in the table below. Retained candidates refers to the average number of candidates retained after pruning, counting mentions with only one retained candidate as one.
The correctness analysis shows that our method retains more alternatives when the top-ranked candidate is incorrect but the gold is in the initial set, suggesting that it adjusts the candidate set sizes in response to the top-ranked candidate correctness. Nonetheless, the gold entity is pruned for approximately 45% of these mentions, and harms exceed rescues, revealing a challenging accuracy–efficiency trade-off. On the other hand, the confidence analysis shows that pruning maintains or improves end-to-end accuracy relative to No-Pruning across all evaluated confidence groups.
We have added the details of this analysis to Appendix D.2 of our revised manuscript. We believe this revision further enhances the analytical depth of our work.
| Group | Mentions | Retained Candidates | Gold Pruned (%) | No-Pruning Acc. | Ours Acc. | Rescue | Harm |
|---|---|---|---|---|---|---|---|
| Correctness | |||||||
| Top-ranked correct | 21,608 | 1.34 | 0.0 | 95.6 | 97.7 | 448 | 0 |
| Top-ranked incorrect, gold in initial set | 3,765 | 5.03 | 45.7 | 33.2 | 27.3 | 88 | 309 |
| Gold absent from initial set | 1,902 | 11.19 | — | 0.0 | 0.0 | 0 | 0 |
| Confidence | |||||||
| Low | 14,918 | 3.81 | 11.3 | 66.6 | 68.1 | 522 | 300 |
| Medium | 8,024 | 1.00 | 2.7 | 95.8 | 95.8 | 14 | 9 |
| High | 4,333 | 1.00 | 0.6 | 99.1 | 99.1 | 0 | 0 |
We hope that our response has addressed your concerns. By conducting additional analyses and revising the manuscript, we believe the manuscript's soundness and analytical depth have been substantially improved, further strengthening our work. If you have any further questions, we would be happy to address them. Thank you again for your time and consideration.
Best,
Authors
To authors
Thank you for the detailed response and additional analysis. The new examples help clarify how the reranker improves selection accuracy by distinguishing correct entities from highly ranked but semantically misleading candidates. The supplementary case studies in Appendix D.5 also provide useful evidence supporting this explanation. Although a more systematic quantitative analysis would further strengthen the findings, the response adequately addresses my main concerns. I therefore maintain my original rating (Weak Accept).
Response to Reviewer JzzX
Dear Reviewer JzzX,
Thank you very much for your reply.
We are glad to hear that our response has adequately addressed your concerns. We also appreciate that you are leaning towards acceptance of our work, with your rating being Weak Accept.
However, we noticed that your reply appears to refer to the points discussed in our response to Reviewer 2 (N81Y). Could we kindly confirm whether this means that you also found our response to Reviewer N81Y's concerns satisfactory, in addition to our response to your own concerns?
Also, we noticed that your initial rating is still listed as 2 (Weak Reject). Given your indication that your current assessment is Weak Accept, we would greatly appreciate it if you could update your rating to 3 (Weak Accept).
If you have any further questions or concerns, please do not hesitate to let us know. We will make every effort to address them before the rebuttal deadline.
We sincerely appreciate your time and consideration.
Best,
Authors
Official Comment by Authors (1/2)
Dear Reviewer JzzX,
We would like to thank you for carefully reviewing our manuscript and for your valuable comments. We particularly appreciate your recognition of key contributions of our work, including efficiency and accuracy gains brought by our method, and our comprehensive evaluation and analysis. Below, we address the questions and concerns raised in your review.
Thank you for pointing this out. Figure 3 describes the average number of candidates passed to the cross-encoder. We have revised Figure 3 and its caption to clarify this.
Thank you for this comment. To further investigate the accuracy–efficiency trade-off beyond configurations selected based on the development-set accuracy, we have conducted a post-hoc analysis comparing our method with the baselines, including additional ones described in Appendix D.6 of the revised manuscript, at similar candidate set sizes passed to the reranker.
Using the same trained pipeline and initial candidate sets, we sweep the hyperparameters of our method and the baselines, including k=1,…,30 for Tuned-Fixed-k and buffer size B=0,...,29 for Adaptive-k. For each setting, we plot accuracy against the micro-average number of candidates passed to the cross-encoder.
Figure 5 of our revised manuscript shows that our method generally achieves higher accuracy than the baselines at comparable numbers of candidates. Top1-Top2 Gate, which is one of the additional baselines, approaches the accuracy of our method that uses the threshold selected on the development set, only by requiring substantially more candidates. These results indicate that our method achieves a superior accuracy–efficiency trade-off across the evaluated settings, compared to the baselines.
We have added the details of these analyses to Appendix D.7 of our revised manuscript.