- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents KNOT, a model that predicts how much a QA model's answer will change when some of its supplied knowledge is removed. It learns from earlier tests in which knowledge was removed and the answers were compared. KNOT considers both the knowledge taken away and what remains, then uses the same model to score groups of knowledge and individual items. The experiments cover five QA benchmarks, knowledge ranking, and context compression. The authors report better results than the compared methods that make no extra calls to the QA model on several measures, but not on every measure or setting.
Critical strengths
-
The problem is useful. Looking at one knowledge item at a time can miss the effect of removing several items together. KNOT offers a way to estimate these effects without repeatedly asking the QA model to answer the question during scoring.
-
The experiments examine more than prediction accuracy. They also test knowledge ranking, what happens when selected knowledge is removed or kept, and whether shorter inputs preserve answer quality. The authors include checks of the training labels and comparisons with simpler model designs. Appendix D.2 tests two more QA models, with a separately trained estimator for each.
Critical weaknesses
-
It is unclear whether the test set was also used to choose the model design. Appendix D.6 and Figure 9 compare model sizes on Ordinary Test500 and use the results to justify six factors and three layers. If these choices were made after seeing the test results, performance on the same test set may look better than it would on new questions. The authors should explain when and how the final design was chosen.
-
Some comparisons need more explanation. Appendix C.1 adds up the individual scores from FastSHAP and Stochastic Amortization to predict a group's effect, while KNOT is trained to predict that effect directly. LOO and Monte Carlo methods also use a different way to measure answer changes. Table 2 provides a useful comparison under shared training conditions, but KNOT does not win on every measure. These results support the method's usefulness, while leaving some uncertainty about what causes its gains.
-
The advantage is smaller when the knowledge comes from the separate retrieval setup. In Table 16, KNOT's ranking score (NDCG@3) is 0.549, compared with 0.541 for simple word-overlap ranking. Word-overlap ranking also has higher Drop scores. The authors acknowledge this mixed result. It limits how broadly the stronger results from the main knowledge setup can be applied, and the size of the ranking difference needs an uncertainty estimate.
4.The number of questions used for the compression results needs clarification. Table 18 reports 77.96%, 78.16%, and 79.16%, which match 389/499, 390/499, and 395/499. The experiment is described as Test500. Please state the actual question count and calculation. Table 10 also lists one training seed, so it is hard to judge how much the main results vary between training runs.
Questions for authors
-
How did you choose six factors and three layers? Did the Test500 results affect that choice? If so, could you check the final model on a separate test set that was not used to choose these settings?
-
LOO and Monte Carlo use a different score for answer changes. If you use the score in Eq. 7 for these methods too, do the main comparisons with KNOT change?
-
In Table 16, KNOT scores 0.549 and word-overlap ranking scores 0.541. Could you report a confidence interval for this difference, so we can judge how reliable the small gain is?
-
How many questions were used in Table 18, and how was accuracy calculated? The percentages seem to match 499 questions rather than 500. Please give the correct-answer counts and explain whether any questions were excluded or the results were averaged in another way.
-
How long does KNOT take to score a new question, including text encoding? What hardware is used? Please also report the time and computing cost of collecting the training data and training KNOT.