acceptodds
Under review as a conference paper at ICLR 2027

SEEKA: Self-Evolving Skill Optimization for Entity Disambiguation over Knowledge Base

Abstract

Entity disambiguation links an ambiguous mention to the entity it denotes in a knowledge base (KB). Existing learned rerankers require labeled data and retraining as domains or entity inventories change, while zero-shot large language model (LLM) rerankers avoid training but cannot improve from recurring errors when their instructions remain fixed. We introduce SEEKA, a nonparametric adaptation framework that co-evolves a KB-grounded synthetic curriculum and the external skill of a frozen LLM reranker (also called Solver). A Proposer generates challenging cases from target entities and their nearest confusers, while an audit verifies factual grounding and unique identification. The Solver’s errors drive validation-gated updates to its instructions, memory, and decision workflow, while a learnability objective adapts the curriculum to the evolving Solver. Across the official held-out test sets of four benchmarks and four Solver families, SEEKA achieves higher top-1 accuracy than the corresponding zero-shot baseline in all 16 model–dataset comparisons, without modifying model parameters or candidate pools.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Entity Linking, Entity Disambiguation, Knowledge Bases, LLM Reranking, Curriculum Learning, Training-Free Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yfUnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SEEKA adapts frozen LLM entity-disambiguation rerankers by jointly updating a synthetic curriculum and an external skill. A Proposer generates contexts around KB entities and confusers, an LLM auditor filters and refines examples, and an Optimizer revises instructions and other declared skill components using synthetic and development feedback. A target-accuracy objective guides curriculum updates. The reported systems improve aggregate accuracy over their corresponding zero-shot baselines in all 16 model–dataset comparisons, with additional analyses of auditing, curriculum size, optimization cycles, and skill transfer.

Critical strengths
  1. The supervision construction addresses the structure of entity disambiguation. Selecting a target and nearby confusers before generating context makes the intended distinction explicit; excluding the target creates NIL examples within the same framework (§2.2). This provides a concrete way to derive adaptation data from the KB while checking factual support and referential ambiguity.

  2. The evaluation provides broad evidence of system-level benefits. Table 6 reports positive aggregate changes across four benchmarks and four Solvers, including experiments using Llama throughout all roles. Fixed candidate pools make these comparisons informative about reranking improvements. The evidence remains heterogeneous: Appendix A appropriately reports that Llama’s NEL improvement is not statistically significant.

  3. The transfer experiment provides initial evidence that learned artifacts can be reused. On DynamicER, transferring the GPT-5.5-developed skill to GPT-5.6 Terra raises accuracy from 89.66% to 89.90% without re-optimization (Table 5d). Although limited to one transfer setting, this supports the practical value of storing adaptation outside model parameters.

Critical weaknesses
  1. The contribution of adaptive co-evolution is not isolated. SIPDO already couples synthetic-example generation with iterative prompt optimization. SEEKA adds task-specific structure, but its experiments lack a fixed-Proposer control and an optimizer using development feedback alone. Table 5 varies auditing, auxiliary models, and curriculum size; the cycle analysis also increases optimization opportunities. Consequently, these results do not establish whether the learnability-driven Proposer update adds value beyond repeated skill optimization with comparable resources. This leaves a central claimed contribution insufficiently validated.

  2. Several headline gains lack a comparison at matched inference cost. Table 8 discloses eight views for the Grok and Terra NEL systems and 22 views for GPT-5.5 on ShadowLink, versus one baseline view. The latter improves accuracy from 90.92% to 91.66%. These are legitimate system improvements, but their benefit relative to simpler ensembling remains unresolved. View counts help, yet token usage, latency, and total optimization cost are not reported, preventing a convincing assessment of the accuracy–cost tradeoff.

  3. Audit acceptance is not an independent measure of curriculum validity. Appendix C uses the Proposer model for separate audit calls, so the 100% refined pass rate establishes compliance with that judge, without independently establishing factual grounding or unique identification. No independent validity assessment is reported. Table 5a provides limited downstream support: auditing adds six correct predictions in one seed, while remaining below the zero-shot baseline. The authors acknowledge that modest gain; my concern is the stronger interpretation of the validity evidence.

Questions for authors
  1. Does adaptive Proposer updating outperform a fixed Proposer, and does synthetic feedback outperform development-only optimization, when optimization and inference budgets are matched?

  2. Clarification and additional comparison: What are the measured inference and adaptation costs of the multi-view systems, and how do they compare with zero-shot ensembling at the same inference budget?

  3. Were accepted examples assessed independently of the generation/audit model? If so, what factual-grounding and unique-identification error rates were observed?

2 comments

We thank the reviewer for recognizing the KB-grounded supervision construction, the breadth of the evaluation, and the potential for skill transfer. We clarify the scope of skill optimization, the interpretation of the reported gains, and will make these distinctions more explicit in the revision.

  1. Contribution of adaptive co-evolution (W1; Q1)

SEEKA optimizes the external decision strategy of a frozen Solver, including instructions, memory, gates, and workflows. To isolate the contribution of adaptive co-evolution, we will add development-only and fixed-Proposer optimization under matched adaptation budgets and development-data.

  1. Inference budget and source of gains (W2; Q2)

The matched comparisons demonstrate gains from the complete SEEKA system over zero-shot reranking. These benefits extend to single-view systems: all four DynamicER systems improve using a single ranking view. Separately, Grok and Llama improve on ShadowLink using synthesized cases only, without labeled development data. Thus, gains are not confined to multi-view or development-supervised configurations. We will add a generic zero-shot ensemble with a matched ranking-call budget to show that SEEKA’s gains extend beyond inference scaling alone.

  1. Interpretation of audit acceptance (W3; Q3)

The accepted examples were not independently assessed by a separate model or human annotators. The reported 100% refined pass rate measures compliance with the generation/audit model’s criteria; it does not independently establish factual grounding or unique identification. We will clarify this distinction and qualify the curriculum-validity claims accordingly.

Thanks for the response and for clarifying the audit pass rate (Q3). Toning down the curriculum-validity claims accordingly makes sense.

The main attribution and cost questions from my review are still open, so I am keeping my rating at 2:

  1. The fixed-Proposer, dev-only, and matched-budget zero-shot ensemble baselines (Q1, Q2) are only planned for the revision. Without the actual numbers, it is still unclear how much of the gain comes from adaptive co-evolution itself.

  2. Pointing to single-view DynamicER and synthesis-only ShadowLink helps clarify the scope (Q2), yet the actual token usage, latency, and total optimization costs for the multi-view systems are still missing.

Reviewer_wg6dOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Paper Summary

The paper proposes SEEKA, a framework for improving entity disambiguation with frozen LLMs by evolving an external skill rather than updating model parameters. A Proposer generates knowledge-base-grounded examples with confusable candidate entities, an Auditor checks whether the examples are factual and unambiguous, and an Optimizer uses Solver errors to revise instructions, memory, and decision workflows. The curriculum also adapts to the Solver’s current performance, and candidate skill changes are selected using held-out development data. The authors evaluate SEEKA on four entity-disambiguation benchmarks with four Solver families, comparing it with matched zero-shot baselines on top-1 accuracy. They report improvements in all 16 model–dataset comparisons.

Critical strengths

Critical Strengths

  1. The paper studies a practical entity-disambiguation setting with varied sources of ambiguity. Its benchmarks cover NIL prediction, popularity bias, and temporal changes in knowledge bases. This gives the proposed framework a meaningful range of tasks to address.

  2. The evaluation includes matched comparisons across multiple Solver models. Within each baseline–SEEKA pair, the Solver, candidate pool, and test examples are held fixed, and the authors report higher top-1 accuracy for SEEKA in all 16 model–dataset comparisons. This provides evidence that the complete adaptation system can improve on zero-shot prompting.

Critical weaknesses

Critical Weaknesses

  1. The experiments do not isolate the source of the reported gains. Although each SEEKA system is compared with a baseline using the same Solver, some SEEKA configurations also use more ranking views, auxiliary-model calls, development data, external memories, or fusion. For example, certain systems combine up to 8 or 22 ranking views, while the zero-shot baseline uses one. The gains may therefore come from additional inference and supervision rather than the proposed self-evolving process. Without comparisons under matched call, data, and compute budgets, the paper’s central claim is difficult to assess.

  2. The core co-evolution mechanism is not sufficiently distinguished from simpler alternatives. The paper does not establish that adapting the curriculum to the Solver’s competence provides benefits beyond ordinary prompt optimization, a fixed audited synthetic curriculum, external memory, or ensembling. The reported ablations do not cleanly isolate these components under matched budgets. As a result, the contribution appears to be a complex system combination, and its necessity over simpler methods remains unclear.

Questions for authors

Questions For Authors

  1. Evidence affecting the recommendation: Can the authors compare SEEKA with simpler alternatives under matched Solver-call, token, and development-data budgets—for example, a fixed prompt optimization method, a fixed audited synthetic curriculum, and multi-view ensembling? This would help determine whether the reported gains come from the co-evolution procedure itself or from additional computation and supervision.

  2. Evidence affecting the recommendation: What are the results when the Proposer’s curriculum is held fixed while the Solver skill is optimized, and when the curriculum adapts but the skill is not updated? These ablations would isolate the contributions of the two proposed adaptation loops.

  3. Clarification: For each reported system, how many total Solver and auxiliary-model calls are used during adaptation and inference, and how much labeled development data is consumed? Please clarify how these budgets compare with the matched zero-shot baseline.

2 comments

We thank the reviewer for recognizing the practical setting and the benefits of the complete adaptation system. We clarify the scope of skill optimization and will make these distinctions more explicit in the revision.

Source of gains and scope of the learned skill

SEEKA optimizes the external decision strategy of a frozen Solver, including instructions, memory, gates, and workflows. These components are part of the adaptation framework.

The matched comparisons demonstrate gains from the complete SEEKA system over zero-shot reranking. These benefits extend to single-view systems: all four DynamicER systems improve using a single ranking view. Separately, Grok and Llama improve on ShadowLink using synthesized cases only, without labeled development data. Thus, gains are not confined to multi-view or development-supervised configurations.

To further distinguish the contributions of skill evolution, inference budget, and development supervision, we will add zero-shot ensembles with matched inference budgets, together with development-only and fixed-Proposer optimization under matched adaptation budgets. We will report Solver and auxiliary-model calls, token usage, and development-data consumption.

Thanks for your response. I would like to keep the score.

Reviewer_BF6MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper adapts a frozen LLM listwise reranker for entity disambiguation without updating its weights. The learnable object is an external skill: instruction, memory, gates, workflow and optional executable components. Each cycle works as follows: 1) A Proposer samples a KB anchor entity and its m nearest retriever neighbours as confusers. It writes a mention in context, linkable or NIL, and a separate LLM audit checks five criteria: grounding, direct denotation, uniqueness, naturalness and a plausible confuser. 2) An Optimizer revises the skill from Solver errors on this curriculum and on a labeled dev feedback fold. A revision is kept only if it improves a dev selection fold. 3) A piecewise-linear learnability reward around a target accuracy τ updates the Proposer instruction. An optional consolidation stage fuses several ranking "views". Four benchmarks are used: NEL, ShadowLink, TempEL and DynamicER. With four Solvers (Grok 4.3, GPT-5.6 Terra, GPT-5.5, Llama 4 Maverick), SEEKA beats the zero-shot reranker in all 16 cells, by 0.12–1.07 points.

Critical strengths
  1. Clean frozen-model protocol. Within each pair, the Solver, reasoning effort, candidate lists and test examples are identical. The full official test splits are used, with paired exact McNemar tests.

  2. The KB-derived confuser and NIL construction is a sensible, reusable way to generate hard cases. The Appendix example shows the intended difficulty.

  3. Honest reporting of weak and negative results: the audit ablation does not beat zero-shot, curriculum size and cycles are non-monotonic, Llama NEL has p=0.78, Llama regresses on two TempEL New slices, and retrieval ceilings and dev-label use are acknowledged.

  4. The method is evaluated on three different kinds of shift and on both closed and open-weight Solvers.

Critical weaknesses
  1. The gains are very small. SEEKA improves on the zero-shot reranker by 0.12–1.07 points in the 16 cells. Five cells have no evidence of significance: four report no p-value, and Llama-NEL has p=.78. "16/16" is a sign count, and the Improvement column makes these gaps look larger than they are.

  2. The gains cannot be attributed to the method. SEEKA-BASE is always a single zero-shot ranking. Many deployed SEEKA systems are ensembles or use gates fitted on up to 2,800 labeled dev mentions, and the design differs per cell. No baseline is matched on views or dev labels. No ablation removes the synthetic curriculum or the learnability-driven Proposer, even though these are the core of the method.

  3. The controlled study does not support the method. On TempEL 2022 New, 7 of the 9 TEST results in Tab 5(a)/(c) tie with or fall below zero-shot. The best configuration is the one favoured by TEST; VALID favours 500 cases, which give no gain.

Questions for authors
  1. For each Tab 8 system, what is the accuracy of SEEKA-BASE with the same number of views and the same fusion (e.g., RRF over the same 7 permutations)? What is the accuracy of SEEKA-BASE plus the same NIL bias or logistic/tree gate, fitted on the same dev data?

  2. What does the evolved instruction alone achieve with a single view and no memory, gate or fusion? This would separate the effect of skill evolution from that of consolidation.

  3. What does SEEKA achieve with no synthetic data, i.e., with the Optimizer using only dev-set errors? What does it achieve with a static Proposer? Please use the same Optimizer budget in both cases.

1 comment

We thank the reviewer for recognizing the frozen-model protocol, the KB-grounded construction, and the breadth and transparency of the evaluation. We clarify the scope of skill optimization, the purpose of the controlled studies, and the interpretation of the reported gains, and will make these distinctions more explicit in the revision.

1. Magnitude and significance of gains (W1)

The Improvement column measures the fraction of remaining correctable errors recovered within the fixed candidate pool. This is directly relevant to reranking: no reranker can recover a gold entity that retrieval has excluded.

Across the three closed-model Solvers, SEEKA recovers approximately 8–21% of the remaining reranking headroom. Consequently, modest absolute accuracy changes can represent meaningful progress within the portion of the task the reranker can improve. Absolute accuracy and headroom recovery provide complementary information, and both are reported.

Paired exact McNemar tests also establish statistically significant improvements across multiple benchmarks and Solvers. We will clarify the headroom metric’s labeling and report the McNemar result for each comparison in the corresponding table.

2. Source of gains and scope of the learned skill (W2; Q1–Q3)

SEEKA optimizes the external decision strategy of a frozen Solver, including instructions, memory, gates, and workflows. These components are part of the adaptation framework, so instruction-only evaluation captures one component of the contribution.

The matched comparisons demonstrate gains from the complete SEEKA system over zero-shot reranking. These benefits extend to single-view systems: all four DynamicER systems improve using a single ranking view. Separately, Grok and Llama improve on ShadowLink using synthesized cases only, without labeled development data. Thus, gains are not confined to multi-view or development-supervised configurations.

To further distinguish the contributions of skill evolution, inference budget, and development supervision, we will add zero-shot ensembles with matched inference budgets, together with development-only and fixed-Proposer optimization under matched adaptation budgets. We will report Solver and auxiliary-model calls, token usage, and development-data consumption.

3. Interpretation of the controlled study (W3)

Table 5(a)/(c) studies auditing and curriculum-size sensitivity under restricted configurations and fixed budgets; complete-system effectiveness is evaluated in Tables 1–4 and 6.

Auditing improves second-cycle accuracy from 45.59% to 46.00%. This controlled configuration remains below zero-shot. The curriculum-size sweep shows that increasing synthetic-data volume does not reliably improve performance under a fixed Optimizer budget. Validation and test favor different curriculum sizes, and the sweep does not establish a validation-selected optimum. This reinforces the importance of a small held-out validation set for final selection. We will make this distinction and the headline systems’ selection procedures explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.