- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SEEKA adapts frozen LLM entity-disambiguation rerankers by jointly updating a synthetic curriculum and an external skill. A Proposer generates contexts around KB entities and confusers, an LLM auditor filters and refines examples, and an Optimizer revises instructions and other declared skill components using synthetic and development feedback. A target-accuracy objective guides curriculum updates. The reported systems improve aggregate accuracy over their corresponding zero-shot baselines in all 16 model–dataset comparisons, with additional analyses of auditing, curriculum size, optimization cycles, and skill transfer.
Critical strengths
-
The supervision construction addresses the structure of entity disambiguation. Selecting a target and nearby confusers before generating context makes the intended distinction explicit; excluding the target creates NIL examples within the same framework (§2.2). This provides a concrete way to derive adaptation data from the KB while checking factual support and referential ambiguity.
-
The evaluation provides broad evidence of system-level benefits. Table 6 reports positive aggregate changes across four benchmarks and four Solvers, including experiments using Llama throughout all roles. Fixed candidate pools make these comparisons informative about reranking improvements. The evidence remains heterogeneous: Appendix A appropriately reports that Llama’s NEL improvement is not statistically significant.
-
The transfer experiment provides initial evidence that learned artifacts can be reused. On DynamicER, transferring the GPT-5.5-developed skill to GPT-5.6 Terra raises accuracy from 89.66% to 89.90% without re-optimization (Table 5d). Although limited to one transfer setting, this supports the practical value of storing adaptation outside model parameters.
Critical weaknesses
-
The contribution of adaptive co-evolution is not isolated. SIPDO already couples synthetic-example generation with iterative prompt optimization. SEEKA adds task-specific structure, but its experiments lack a fixed-Proposer control and an optimizer using development feedback alone. Table 5 varies auditing, auxiliary models, and curriculum size; the cycle analysis also increases optimization opportunities. Consequently, these results do not establish whether the learnability-driven Proposer update adds value beyond repeated skill optimization with comparable resources. This leaves a central claimed contribution insufficiently validated.
-
Several headline gains lack a comparison at matched inference cost. Table 8 discloses eight views for the Grok and Terra NEL systems and 22 views for GPT-5.5 on ShadowLink, versus one baseline view. The latter improves accuracy from 90.92% to 91.66%. These are legitimate system improvements, but their benefit relative to simpler ensembling remains unresolved. View counts help, yet token usage, latency, and total optimization cost are not reported, preventing a convincing assessment of the accuracy–cost tradeoff.
-
Audit acceptance is not an independent measure of curriculum validity. Appendix C uses the Proposer model for separate audit calls, so the 100% refined pass rate establishes compliance with that judge, without independently establishing factual grounding or unique identification. No independent validity assessment is reported. Table 5a provides limited downstream support: auditing adds six correct predictions in one seed, while remaining below the zero-shot baseline. The authors acknowledge that modest gain; my concern is the stronger interpretation of the validity evidence.
Questions for authors
-
Does adaptive Proposer updating outperform a fixed Proposer, and does synthetic feedback outperform development-only optimization, when optimization and inference budgets are matched?
-
Clarification and additional comparison: What are the measured inference and adaptation costs of the multi-view systems, and how do they compare with zero-shot ensembling at the same inference budget?
-
Were accepted examples assessed independently of the generation/audit model? If so, what factual-grounding and unique-identification error rates were observed?
2 comments
Thanks for the response and for clarifying the audit pass rate (Q3). Toning down the curriculum-validity claims accordingly makes sense.
The main attribution and cost questions from my review are still open, so I am keeping my rating at 2:
-
The fixed-Proposer, dev-only, and matched-budget zero-shot ensemble baselines (Q1, Q2) are only planned for the revision. Without the actual numbers, it is still unclear how much of the gain comes from adaptive co-evolution itself.
-
Pointing to single-view DynamicER and synthesis-only ShadowLink helps clarify the scope (Q2), yet the actual token usage, latency, and total optimization costs for the multi-view systems are still missing.
We thank the reviewer for recognizing the KB-grounded supervision construction, the breadth of the evaluation, and the potential for skill transfer. We clarify the scope of skill optimization, the interpretation of the reported gains, and will make these distinctions more explicit in the revision.
SEEKA optimizes the external decision strategy of a frozen Solver, including instructions, memory, gates, and workflows. To isolate the contribution of adaptive co-evolution, we will add development-only and fixed-Proposer optimization under matched adaptation budgets and development-data.
The matched comparisons demonstrate gains from the complete SEEKA system over zero-shot reranking. These benefits extend to single-view systems: all four DynamicER systems improve using a single ranking view. Separately, Grok and Llama improve on ShadowLink using synthesized cases only, without labeled development data. Thus, gains are not confined to multi-view or development-supervised configurations. We will add a generic zero-shot ensemble with a matched ranking-call budget to show that SEEKA’s gains extend beyond inference scaling alone.
The accepted examples were not independently assessed by a separate model or human annotators. The reported 100% refined pass rate measures compliance with the generation/audit model’s criteria; it does not independently establish factual grounding or unique identification. We will clarify this distinction and qualify the curriculum-validity claims accordingly.