acceptodds
Under review as a conference paper at ICLR 2027

Structured Knowledge Dependency Estimation for Question Answering

Abstract

Question answering (QA) systems draw on knowledge from task context, retrieval, decomposition, and intermediate reasoning. These sources form a candidate knowledge space whose units may be redundant, complementary, noisy, or incorrect, making their influence difficult to assess independently. In particular, the effect of removing a knowledge subset can differ from the sum of its individual parts, while exhaustive subset interventions are costly. We formulate this challenge as knowledge dependency estimation, defining dependency by the behavioral change of a fixed QA model when subsets of candidate knowledge are removed. We propose KNOT, a structured estimator learned from offline interventions. KNOT maps candidates into shared latent factors and uses removal–retention coverage to model both the coverage carried by a removed subset and the coverage preserved by the remaining candidates. The same function predicts subset effects and yields unit-level dependency scores through singleton evaluation. Across five QA benchmarks, KNOT improves zero-call subset-effect prediction and knowledge ranking over amortized attribution baselines, and supports substantial context compression while preserving near-full-context answer quality.

Primary area:
interpretability and explainable AI
Keywords:
Knowledge Dependency Estimation, Question Answering, Structured Coverage Modeling, Knowledge Attribution, Context Compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Mc3QOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents KNOT, a model that predicts how much a QA model's answer will change when some of its supplied knowledge is removed. It learns from earlier tests in which knowledge was removed and the answers were compared. KNOT considers both the knowledge taken away and what remains, then uses the same model to score groups of knowledge and individual items. The experiments cover five QA benchmarks, knowledge ranking, and context compression. The authors report better results than the compared methods that make no extra calls to the QA model on several measures, but not on every measure or setting.

Critical strengths
  1. The problem is useful. Looking at one knowledge item at a time can miss the effect of removing several items together. KNOT offers a way to estimate these effects without repeatedly asking the QA model to answer the question during scoring.

  2. The experiments examine more than prediction accuracy. They also test knowledge ranking, what happens when selected knowledge is removed or kept, and whether shorter inputs preserve answer quality. The authors include checks of the training labels and comparisons with simpler model designs. Appendix D.2 tests two more QA models, with a separately trained estimator for each.

Critical weaknesses
  1. It is unclear whether the test set was also used to choose the model design. Appendix D.6 and Figure 9 compare model sizes on Ordinary Test500 and use the results to justify six factors and three layers. If these choices were made after seeing the test results, performance on the same test set may look better than it would on new questions. The authors should explain when and how the final design was chosen.

  2. Some comparisons need more explanation. Appendix C.1 adds up the individual scores from FastSHAP and Stochastic Amortization to predict a group's effect, while KNOT is trained to predict that effect directly. LOO and Monte Carlo methods also use a different way to measure answer changes. Table 2 provides a useful comparison under shared training conditions, but KNOT does not win on every measure. These results support the method's usefulness, while leaving some uncertainty about what causes its gains.

  3. The advantage is smaller when the knowledge comes from the separate retrieval setup. In Table 16, KNOT's ranking score (NDCG@3) is 0.549, compared with 0.541 for simple word-overlap ranking. Word-overlap ranking also has higher Drop scores. The authors acknowledge this mixed result. It limits how broadly the stronger results from the main knowledge setup can be applied, and the size of the ranking difference needs an uncertainty estimate.

4.The number of questions used for the compression results needs clarification. Table 18 reports 77.96%, 78.16%, and 79.16%, which match 389/499, 390/499, and 395/499. The experiment is described as Test500. Please state the actual question count and calculation. Table 10 also lists one training seed, so it is hard to judge how much the main results vary between training runs.

Questions for authors
  1. How did you choose six factors and three layers? Did the Test500 results affect that choice? If so, could you check the final model on a separate test set that was not used to choose these settings?

  2. LOO and Monte Carlo use a different score for answer changes. If you use the score in Eq. 7 for these methods too, do the main comparisons with KNOT change?

  3. In Table 16, KNOT scores 0.549 and word-overlap ranking scores 0.541. Could you report a confidence interval for this difference, so we can judge how reliable the small gain is?

  4. How many questions were used in Table 18, and how was accuracy calculated? The percentages seem to match 499 questions rather than 500. Please give the correct-answer counts and explain whether any questions were excluded or the results were averaged in another way.

  5. How long does KNOT take to score a new question, including text encoding? What hardware is used? Please also report the time and computing cost of collecting the training data and training KNOT.

Reviewer_nMp2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how much a QA model's answer changes when some candidate knowledge is removed. The authors formalize this as knowledge dependency estimation and train a lightweight estimator on offline removal experiments, so the change can be predicted without querying the target QA model at test time.

Critical strengths
  1. The method is simple and easy to follow, and the problem formulation is clear. This method could be useful for compressing context without hurting the model's answers.
  2. The experiments are extensive. The authors are also transparent about where their method falls short.
Critical weaknesses
  1. Avoiding test-time QA calls is one of the main selling points, but building the training set already takes many calls, and this has to be redone for each target model. Since KNOT is still less accurate than methods that use test-time calls, it would help to clarify when this trade-off is actually worth it.

  2. I'm not fully convinced by the experimental results. Here are my reasons:

    (1) I have some concerns about the labels. Figure 7a shows that the NLI score can change a lot just from option prefix differences, so the labels may be fairly noisy. KNOT is trained directly on this target, while LOO and MC use a different utility, which makes the comparison in table 1 a bit uneven. Even so, on ordinary test, size-only ridge gets a higher spearman than KNOT, and KNOT trails LOO, MC-S4, and MC-S16.

    (2) The test set has only 500 questions, the results come from a single seed, and there are no confidence intervals, while many of the gaps are small. It might be helpful to report bootstrap CIs, and ideally results over a few seeds.

Questions for authors
  1. The factor activations a_{ir} are computed once on the full candidate set C(q) and reused for every removal subset S, so the retained candidates are still conditioned on the removed ones through the set encoder's attention. For example, if the encoder down-weights k_2 because it overlaps with k_1, removing k_1 could overestimate the loss, even though k_2 can still serve as a substitute. Why not re-encode C(q) \ S instead?

  2. In the ordinary setting, the model can also rely on its own parametric knowledge. What is the model's answer accuracy when given only the question, without any candidate knowledge?

  3. Does a trained estimator transfer to a new target model without retraining? Can it generalize across models?

Reviewer_v3grOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates "knowledge dependency estimation" for QA as predicting the behavioral change of a fixed QA model when subsets of candidate knowledge are removed. It proposes KNOT, which encodes candidates with a frozen LLM encoder and a Set Transformer, maps them to latent factors, and predicts subset effects through a noisy-OR "removal–retention coverage" function. Unit-level scores come from singleton evaluation of the same function. Experiments on five QA benchmarks show lower zero-call error and better ranking than amortized baselines, plus modest context-compression utility.

Critical strengths
  1. Clear and well-motivated problem formulation. Defining dependency behaviorally at the subset level, and noting that removal effects are non-additive under redundancy, is a sensible and practically relevant framing.
  2. Broad and fairly transparent evaluation. It covers five benchmarks, three evaluation conditions, baselines with and without test-time calls, cost-normalized comparison (MAE gain per call), two extra target models, ablations, and capacity scans.
  3. Practical angle. Zero-test-time-call scoring and the compression use case are relevant to RAG and long-context settings.
Critical weaknesses
  1. Results use one seed (42) and 500 test questions, with no confidence intervals or significance tests.
  2. The central claim, that KNOT models set-level interactions, is not directly validated. There is no controlled test of redundancy or complementarity. There is no measurement of predicted vs. true non-additivity, no error stratification by subset size or redundancy, and no analysis of what the factors capture.
  3. The weight α=0.4 is arbitrary and has no sensitivity analysis.
  4. Candidates (decomposition, reasoning, option-discrimination — 33% of MMLU units) are generated by the same Qwen3.5-35B target model, creating a risk of circularity or answer leakage.
  5. Compression claims are overstated. KNOT's advantages are 0.5–1 pp on 500 questions.
Questions for authors
  1. Was Ordinary Test500 used to select R and L (Fig. 9)? Why does the 1,000-question model (MAE .190) beat the 5,000-question model (MAE .220) on the same test set? Please provide a learning curve and a clean held-out selection.
  2. What direct evidence shows that KNOT captures redundancy and complementarity?
  3. Can the unit-scoring baselines be fairly calibrated (isotonic or linear on dev) for the MAE comparison?
  4. Please provide a total-cost analysis, including offline supervision calls, training, encoder inference, and break-even versus LOO. How well does KNOT transfer to a new target model or candidate source without retraining?
  5. For compression, please add confidence intervals, a no-context baseline, latency/total-token savings, and evaluation on more realistic RAG candidate sets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.