- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies question answering over incomplete knowledge graphs where facts carry extra details, such as the work an award was given for. The authors introduce FIHR-CQA to select questions that require predicting at least two distinct missing facts and remove examples with specified shortcuts. They also propose HyWit, which predicts candidate facts and combines those that agree on shared entities. It learns which additional candidates to generate within a fixed budget and returns both an answer and its supporting facts. On the filtered WD50K-NFOL dataset, HyWit gets both the answer and its supporting facts right in 21.4% of cases, compared with 19.7% for the strongest baseline best-first search.
Critical strengths
- The filter checks every possible way to answer a question in the full graph, not just one sampled path. This is a good setting as a different path might reach the same answer with fewer missing facts. The check for distinct missing facts removes 3,733 examples that passed the earlier check, so this distinction affects a large part of the dataset.
- One ablation directly shows that training on supporting facts helps. With the candidate facts held fixed, removing this training objective lowers joint answer-and-evidence accuracy from 21.4% to 18.5%, while candidate coverage and answer accuracy stay the same. This shows the gain in better selection of the correct supporting facts.
Critical weaknesses
- The practical benefit over the strongest baseline remains unclear. With the same proposer and scorer backbones, HyWit improves joint recovery over best-first search only from 19.7% to 21.4% (Table 1), and a simple deficit heuristic already reaches 18.3%. Coverage is only 44.8%, so for most questions the correct supporting facts are never generated, and better acquisition alone cannot fix this. In the ablations (Table 2), the largest drop comes from removing the evidence loss, not the learned acquisition, and the remaining differences are small and reported without variance. The runtime, transfer, and masking experiments compare only with Static, so it is unclear whether HyWit beats best-first search at the same runtime or in harder settings. Key details are also missing: the proposer and scorer architectures, the actual budget sizes, and how evidence is extracted for the baselines. This makes the results hard to reproduce and the comparisons hard to judge.
- Figure 3 shows that the new checks remove many questions, but not how this changes model performance. Prior work already filters questions so that they require predicting multiple missing links; this paper instead requires multiple missing statements. To show why this difference matters, the authors should compare model performance before and after the new checks on questions with similar structures.
Questions for authors
- Could you compare HyWit with best-first search at the same runtime and on unseen query structures? What is the additional cost of training the controller?
- Do the search baselines (best-first, beam, and the deficit heuristic) use the same scorer trained with the evidence loss? If not, how much of HyWit’s improvement remains when they do?
- Could you provide the proposer and scorer architectures, the actual budget settings, and the evidence-extraction procedure used for the baselines?
- How does model performance change after the new filtering checks, when comparing questions with similar structures?
Response to Reviewer u2BM (Orthogonal Ablation, BF across RQ4–6, Audit Strata & Specs)
Dear Reviewer u2BM,
We sincerely apologize for the late rebuttal due to urgent work commitments. We have uploaded the revised PDF with substantial new experiments, full architectural and protocol disclosures, and ten-seed statistical testing. We would be deeply grateful if you could spare time to read our updated manuscript. Below we directly address every technical concern you raised.
1. Margin Over Best-First Search & Whether Search Controls Share
L_evThank you for highlighting this crucial question. In Table 1, all four search controls (Static Acquisition, Deficit Heuristic, Beam Search, and Best-First Search) already share our exact StarE-Transformer proposer and
L_score-trained (i.e.,L_ev-supervised) scorer (Section 5.1 and Appendix F). Thus, Table 1's +1.7 pp (WD50K-NFOL) and +1.9 pp (WikiPeople) margins over Best-First isolate only the learned acquisition controller. To show the full contribution transparently, we added a complete ten-seed 2×2 orthogonal ablation (Figure 3(c) and Appendix D.1, Table 3):L_ev), Full HyWit improves strict J@1 by +4.3 pp (+25.1% relative) on WD50K-NFOL (17.1% → 21.4%) and +4.2 pp (+26.9% relative) on WikiPeople (15.6% → 19.8%, exact paired sign-flip p = 0.0020).L_evand learned acquisition exhibit statistically significant synergy (+1.0 pp on WD50K, +1.1 pp on WikiPeople, p = 0.0020).2. Adding Best-First Search & Deficit Heuristic to RQ4–RQ6 + Ten-Seed Variance
We upgraded all experiments across RQ1–RQ6 to include Best-First Search and Deficit Heuristic alongside Static Acquisition, with ten paired seeds (mean ± std) and exact two-sided paired sign-flip p-values (Table 1, Figures 3, 5, 6, 7, Appendix D):
3. Diagnosing the 44.8% Coverage Bottleneck
In Section 5.6 (Figure 6(b)–(c)) and Appendix D.4 (Table 6), we decompose HyWit's 44.8% covered targets into 47.8% Valid Joint Recovery (21.4% abs.), 19.6% Intra-Entity Spurious Witness Override (8.8% abs., reduced from 26.1% w/o
L_ev), and 32.6% Inter-Entity Hub Distractor Competition (14.6% abs.). We also added four LLM/Graph-R1 agent baselines (Figure 7(c)), where HyWit leads Graph-R1 by +5.5 pp J@1 at 97.1% lower latency.4. Pre- vs. Post-Audit Performance Comparison
We added a stage-by-stage evaluation across FIHR-CQA audit strata on structurally matched three-factor queries (Section 5.4, Figure 4(c); Appendix D.2, Table 4). Observed Projection Lookup drops from 64.2% H@1 on projection shortcuts (
P_Γ(q)) to 0.0% on strict targets, and every learned model drops by >20 pp H@1 once single-record self-joins (D(q)) and projection shortcuts are removed, empirically validating the necessity of statement-level auditing.5. Complete Architecture, Budget, and Baseline Extractor Specifications
We now fully specify all components in Section 5.1, Appendix C (Algorithm 1), Appendix F, and Appendix H: the 2-layer StarE encoder (d = 256) + 4-layer, 8-head Transformer decoder (beam b = 16), 2-layer Transformer scorer, 2-layer 64-unit MLP controller, exact budget parameters (B = 72,
B_init = 18, k = 6, r = 4,D_max = 3, H = 18), and the shared reference-blind post-hoc witness extractor used for NQE and StarQE.We would be truly grateful if you could spare time to read our revised PDF. If these comprehensive additions and clarifications resolve your concerns, we kindly ask you to consider raising your score. Thank you for your thorough feedback!