- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CoRE, a framework for structured knowledge retrieval that jointly learns a rule-routing policy and rule-constraining retrievers without explicit query-rule supervision. The key idea is to treat the reasoning rule as a latent variable and optimize routing and retrieval through alternating bi-level updates driven by downstream retrieval feedback. Experiments on three STaRK-QA datasets show that CoRE outperforms several structured and agentic retrieval baselines, and ablations highlight the benefit of co-adaptation.
Critical strengths
- The problem of learning rule routing without query-rule labels is well motivated and practically relevant.
- The co-adaptive optimization framework is conceptually clean and supported by ablation studies.
- The case study in Appendix C illustrates the gap between semantic rule plausibility and retrieval effectiveness.
Critical weaknesses
-
Insufficient treatment of the rule set itself.
The rules are predefined from the graph schema (6 for AMAZON, 8 for MAG, 65 for PRIME). The paper does not study rule-set quality, rule-number sensitivity, redundancy, or automatic rule discovery/pruning. Appendix B already shows that the rule space is over-complete and contains many tied-optimal or rarely useful rules, yet the main experiments never ablate the rule set. This is a major scalability concern: if rules must be hand-crafted and the method is sensitive to their quality and size, it is difficult to apply to real, evolving knowledge graphs. -
Reward design is coarse.
The routing reward is reciprocal rank of the ground-truth evidence under a given rule, using the best rank among correct answers for multi-answer queries. It does not directly optimize recall, multiple-answer coverage, score margins, or retrieval cost. A more informative reward (e.g., NDCG or a recall-aware composite) could substantially change routing behavior, but this is not explored. -
Error analysis is limited.
Appendix C provides a confusion matrix and two representative case studies, but there is no systematic breakdown of routing errors vs. retrieval errors, no analysis of which query types or rules fail, and no mechanism to detect or recover from routing mistakes. This limits our understanding of the method’s failure modes.
Questions for authors
- How sensitive is CoRE to the number and quality of rules? Can redundant rules be pruned automatically?
- Would a richer reward (e.g., NDCG or recall-aware) improve routing?
- Can the authors provide a quantitative error analysis separating routing errors from retrieval errors?
5 comments
[Q1] Sensitivity to rule number, quality, and redundancy - (Part2)
4. Automatic rule discovery
We also implemented an LLM-assisted discovery procedure to investigate whether retrieval feedback can guide discovery of more useful rules, which will subsequently lead to more effective retrieval.
- Identify queries for exploration. We collect training queries for which none of the existing rule-conditioned retrievers ranks a correct answer as top 1. These failures identify opportunities for rule set expansion.
- Propose rules. We prompt an LLM with the graph schema, five sampled queries from this set, and their ground-truth answer entity types, asking it to propose one potentially useful retrieval rule.
- Validate proposals. We check graph validity and reject rules that duplicate the current rule set. We repeat this process until 65 additional rules are collected.
- Retrain and evaluate. We combine the original and proposed rules and retrain CoRE with the expanded 130-rule set.
| Rule-space setting | Rules | H@1 | MRR |
|---|---|---|---|
| Original | 65 | 49.34 | 61.27 |
| Expanded | 130 | 51.79 | 63.34 |
The expanded rule set improves both H@1 and MRR. This supports the potential of using training-time retrieval failures to guide rule proposals and of CoRE to exploit the resulting alternatives. It also provides a practical way to expand the rule set without manually designing each additional rule.
Together, these experiments show how retrieval feedback can inform both which rules to retain and where to explore new rules. Pruning offers a more compact rule set with lower retrieval workload, while discovery improves retrieval performance by introducing additional alternatives.
These experiments and their construction details are included in Section 5.4 and Figure 3 of the revised paper.
[Q2] Reward design and multi-answer coverage
Reciprocal rank (RR) emphasizes the earliest correct answer without directly rewarding additional relevant answers. We evaluate three alternative routing rewards on PRIME and MAG.
- Alternative rewards. We compare the original RR reward with NDCG@20, Recall@20, and 0.5 RR + 0.5 Recall@20, keeping initialization, data splits, training schedules, and budgets consistent.
- Evaluation. We report H@1 and MRR to assess early ranking quality, Recall@20 to measure answer coverage, and NDCG@20 to assess the ranking of relevant answers within the retrieved list.
PRIME
| Routing reward | H@1 | MRR | Recall@20 | NDCG@20 |
|---|---|---|---|---|
| RR (original) | 49.34 | 61.27 | 80.81 | 61.84 |
| NDCG@20 | 49.48 | 62.15 | 82.23 | 62.97 |
| Recall@20 | 49.16 | 61.99 | 82.47 | 62.88 |
| 0.5 RR + 0.5 Recall@20 | 48.75 | 61.63 | 82.03 | 62.49 |
MAG
| Routing reward | H@1 | MRR | Recall@20 | NDCG@20 |
|---|---|---|---|---|
| RR (original) | 63.37 | 74.17 | 83.51 | 71.74 |
| NDCG@20 | 66.26 | 75.66 | 84.75 | 72.38 |
| Recall@20 | 62.14 | 72.68 | 83.77 | 70.04 |
| 0.5 RR + 0.5 Recall@20 | 66.34 | 75.56 | 83.83 | 71.87 |
Overall, richer rewards improve answer coverage on both datasets, with NDCG@20 providing the most consistent gains across ranking and coverage metrics. Recall-only rewards favor coverage but can sacrifice early ranking quality, while combining RR and recall offers a balance whose effectiveness varies by dataset. These results support the reviewer’s suggestion and demonstrate that CoRE can accommodate different retrieval priorities through its reward design.
The reward definitions and comparative results are included in Appendix H, Table 5 of the revised paper.
[Q3] Quantitative error decomposition and failure modes
We have added an error breakdown across all three datasets and a routing-correction experiment on MAG.
1. Separating routing errors from retrieval errors
We divide queries into three outcomes:
- Successful retrieval: the selected rule ranks a correct answer as top-1.
- Routing error: the selected rule fails, but another rule ranks a correct answer as top1.
- Retrieval error: no rule ranks a correct answer as top1, so changing the routing decision alone would not resolve the failure.
Percentages below are calculated over all queries evaluated in this analysis.
| Dataset | Queries | Successful retrieval | Routing error | Retrieval error |
|---|---|---|---|---|
| AMAZON | 1,719 | 50.77% | 16.50% | 32.73% |
| MAG | 2,748 | 63.37% | 8.35% | 28.28% |
| PRIME | 2,899 | 49.52% | 21.37% | 29.11% |
The results show that both routing and retrieval contribute to the remaining errors. PRIME has the largest proportion of failures recoverable through a different rule choice, while MAG has the smallest. Across all three datasets, however, a substantial portion of failures persists even with the best available rule.
2. Can routing mistakes be recovered?
To investigate this possibility, we introduce a lightweight routing correction model on MAG. Specifically, the model takes the frozen BERT query embedding and CoRE’s predicted routing distribution as input, and learns to produce a corrected routing distribution using retrieval rewards from training queries as supervision.
At inference, we compare the highest-ranked rule from the original and corrected distributions. If they agree, the original routing decision is retained; otherwise, the rule with the highest corrected score is selected. The original data splits and trained retrievers remain unchanged, and only the final selected retriever is executed.
We evaluate the correction mechanism using the original evaluation protocol.
| Method | H@1 (%) | MRR |
|---|---|---|
| CoRE | 63.37 | 0.7417 |
| CoRE + correction | 64.77 | 0.7534 |
The improvement in both H@1 and MRR demonstrates that routing decisions can be refined using information already available in the query representation and original routing distribution.
Together, these results demonstrate the feasibility of detecting and correcting routing errors without modifying the underlying retrievers. We include these analyses in Appendix E, Figure 10, and Table 3 of the revised paper.
We thank the reviewer for recognizing CoRE's motivation and co-adaptive framework, and for the constructive suggestions on rule-set design, reward choice, and failure analysis. In response, we have added rule-set sensitivity, pruning, and discovery experiments; compared alternative routing rewards; and analyzed routing versus retrieval errors, together with a correction experiment.
[Q1] Sensitivity to rule number, quality, and redundancy - (Part1)
We have added experiments on PRIME examining rule-number sensitivity, rule-quality sensitivity, automatic pruning, and rule discovery.
1. Sensitivity to rule number
We compare randomly sampled subsets of 20, 30, 40, and 50 rules with the full 65-rule set.
Performance generally improves as more rules become available. Subsets of 40–50 rules retain much of the full set's answer coverage, while the complete rule set provides the strongest ranking performance. This suggests that additional rules offer useful retrieval alternatives, but their contribution depends on which rules are included.
2. Sensitivity to rule quality
We measure rule utility using the fractional oracle frequency introduced in Appendix B, Figure 6. For each query, a rule receives one unit of credit if it uniquely achieves the highest retrieval reward, or 1/k credit if it ties with k−1 other rules. A rule's utility score is its total Unique + Tied credit, normalized by the number of queries. Thus, the score reflects how frequently a rule provides retrieval-optimal results while accounting for equally effective alternatives.
We rank rules by this score and compare equal-sized subsets from different positions in the ranking.
At a fixed rule-set size, higher-utility subsets consistently perform better across all three metrics. The top 10 rules even outperform the random 20- and 30-rule subsets, indicating that rule utility matters beyond rule count. Nevertheless, the full rule set remains stronger than the top 10, suggesting that rules with lower overall oracle frequency can collectively provide complementary value for queries less well served by the highest-ranked rules.
3. Automatic pruning
Motivated by these findings, we evaluated a simple usage-based pruning strategy that retains the 30 most frequently used rules and removes the remaining 35. The following table reports the reduced rule set alongside its full-set reference in this pruning experiment.
Removing more than half of the rules retains performance close to the full-set reference. We also quantify the associated changes in model size and computational workload:
These results demonstrate the benefit of pruning unnecessary rules in maintaining retrieval performance and reducing resource complexity, which provides a practical starting point for future exploration on rule-aware pruning.