- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies evidence selection for retrieval-augmented question answering with a limited context budget. It focuses on a limitation of leave-one-out attribution: when multiple passages provide the same information, removing one may not affect the answer, so useful passages can receive zero credit. The authors use Banzhaf values to estimate the contributions of passages across different subsets of retrieved evidence. They train CAMS, a small setwise selector operating on frozen embeddings, to predict these values without calling a language model during selection. Experiments examine evidence redundancy, downstream answer quality, and the effects of credit-based versus label-based supervision.
Critical strengths
-
The paper demonstrates the limitation of LOO attribution under redundant evidence using a simple two-copy OR game and an exact-copy stress test. The downstream evaluation also reports paired confidence intervals for the performance improvements.
-
The C/L/B comparison controls for architecture, input features, candidate pools, and random seeds. This allows the effects of different supervision targets to be compared under consistent experimental conditions.
-
The statistical reporting includes paired bootstrap confidence intervals, standard deviations across seeds, and an explicit distinction between exploratory and confirmatory analyses.
Critical weaknesses
-
The practical advantage of CAMS over simpler alternatives is limited. The main improvement is approximately 0.04 F1 on one candidate pool at one context budget. Meanwhile, a standard BGE cross-encoder, requiring 269 ms per query, substantially outperforms CAMS under tighter budgets. On Hard-100 with k=3, the cross-encoder achieves 0.665 F1 compared with 0.586 for CAMS. The paper does not establish when CAMS should be preferred over this baseline.
-
The claim that credit supervision can replace labels is not supported by an appropriate statistical test. The conclusion relies on 21 of 22 comparisons showing no statistically significant difference, but non-significant differences do not establish equivalence. No equivalence margin is defined, and the only statistically significant comparison favors label supervision.
-
The evaluation uses a single dataset with injected redundancy, and the improvement on the realistic candidate pool does not transfer to the second reader. For Llama, CAMS achieves 0.623 F1 compared with 0.627 for dense retrieval. The offline training requires 6.1 million reader calls and 68.8 GPU-hours, but the paper does not analyze whether this cost is justified by the online improvements compared with directly using a cross-encoder.
Questions for authors
-
What motivated the choice of Banzhaf rather than Shapley values? Were both tested as supervision targets?
-
The credit-versus-label claim would benefit from an equivalence test with a predefined margin. Can the authors provide this analysis?
-
Given the 6.1 million offline reader calls, approximately how many online queries would be needed for CAMS to offset its training cost relative to the BGE cross-encoder?
2 comments
Response on Banzhaf vs Shapley, Feedback Statistics, and CE Cost-2
Motivation for Banzhaf Values and Direct Student Comparison with Shapley Values
-
We use Banzhaf values because they weight all passage subsets uniformly, matching uniform subset sampling, whereas Shapley weights depend on subset size and require different estimators. We directly trained and compared student selectors on Banzhaf versus Shapley targets across 20 settings under both matched sampled coalition counts and full enumeration.
-
Under matched sampled coalition counts across the 20 settings, 19 paired difference confidence intervals crossed zero and 1 favored Banzhaf. Under full enumeration across the 20 settings, 3 confidence intervals favored Shapley and 17 crossed zero.
-
Across 20 student-level settings, Banzhaf and Shapley targets yield statistically indistinguishable Answer F1 in 19 of 20 sampled settings (with 1 favoring Banzhaf) and 17 of 20 full-enumeration settings (with 3 favoring Shapley).
Comparison with Cross-Encoder (CE), Feature Fusion, and Speed-Quality Trade-off
-
Building on the CE feature fusion in our manuscript, we evaluated CAMS with CE features under our newly matched Hard-100 k=8 setup, and timed full retrieval-to-answer execution across 40 questions each on HotpotQA pools, MuSiQue pools, and a single-H20 preloaded 32.5M-passage KILT setup with known-question candidate caching.
-
On Hard-100 at k=8, pure CAMS scored 0.6486 Answer F1, CAMS+CE fusion scored 0.6955 (+0.0469 over pure CAMS), and full CE scored 0.7036 (0.0081 above fusion). In the 40-question end-to-end runs, CAMS had lower median latency and lower F1 than CE on HotpotQA (1.82s vs. 2.05s; F1 0.496 vs. 0.616, so CAMS is 0.23s faster and 0.120 F1 lower) and MuSiQue (2.174s vs. 2.306s; F1 0.2774 vs. 0.3268, 0.132s faster and 0.0494 F1 lower). On KILT, median latencies were 3.749s (Dense), 3.960s (CAMS), and 3.982s (CE).
-
Adding CE features closes 85% of the F1 gap between pure CAMS and full CE on Hard-100 k=8, while pure CAMS provides a faster, lower-F1 operating point than full CE.
Offline Cost Arithmetic, New Generation Timing, and Cross-Dataset / Reader Scope
-
We computed a conditional arithmetic ratio of historical offline cost (6.14M calls, 68.8 GPU-hours) to selection-stage latency savings vs. CE, timed fresh 1,500-question teacher generation (8,363,968 responses across 128, 256, and 1,024 budgets), and tested MuSiQue (250 pool and 250 KILT questions) and 512-token Llama-3.1-8B-Instruct.
-
The arithmetic ratio equals ~2.76 million queries under the formal assumption of identical hardware, utilization, and quality (whereas CE has higher F1). For new teacher generation, we report only summed reader-call durations: 53.11, 106.12, and 423.85 hours at 128, 256, and 1,024 budgets. On MuSiQue at k=3, in-domain CAMS scored 0.3689 F1 (vs. 0.3114 Dense and 0.4488 CE), while zero-shot transfer (0.3260) and KILT retrieval (0.1414 vs. 0.1290 Dense) had CIs crossing zero. Retraining Llama at 512 tokens gave 22 deduplicated CIs crossing zero; running Llama on fixed E4 selections (26,818 safe / 28,076 strict responses) tested budget effects on another reader (Hard-100 k=8 strict 128−1024 diff −0.0061).
-
Fresh teacher generation takes 53.11 and 106.12 summed call-hours at the smaller budgets, compared with 423.85 at 1,024. In-domain MuSiQue CAMS exceeds Dense by 0.0575 F1. The KILT and zero-shot CAMS-minus-Dense intervals include zero, while the Llama length-control rerun has 22 zero-containing intervals.
Thank you for helping us sharpen our statistical and practical analysis.
Response on Banzhaf vs Shapley, Feedback Statistics, and CE Cost
Thank you for your time and thorough review. We will respond to your comments point by point and detail the new experiments conducted. Addressing your questions on choosing Banzhaf over Shapley, the statistical basis for label substitution, cross-encoder quality and efficiency, and training costs, we performed a direct comparison with Shapley values, statistical equivalence testing between credit and labels, cost-quality analyses across new budgets, cross-encoder fusion experiments, and end-to-end timing measurements. Below, we explain the procedures, results, and corresponding paper conclusions for each point.
Equivalence Test Between Coalition Feedback and Passage Labels vs. Budget Comparisons
Following Table 2 (where 21 of 22 comparisons were non-significant), we compared CAMS trained with coalition feedback (CAMS-C) against supporting-passage labels (CAMS-L) on 1,000 additional held-out questions on Hard-100 at k=8 using a two-sided ±0.02 Answer F1 equivalence bound.
Coalition feedback achieved 0.6620 Answer F1 versus 0.6473 for passage labels, outperforming labels by +0.0147 F1 with a paired 90% confidence interval of [+0.0059, +0.0236]. The ±0.02 two-sided equivalence condition was not met because coalition feedback outperformed labels enough to push the upper confidence bound (+0.0236) above +0.02. Separately, in our E4 strict-target budget study, 37 of 44 non-deduplicated Qwen comparisons fell within an exploratory ±0.02 margin, which measures reduced budgets (128/256 vs. 1,024) rather than credit-vs.-label supervision.
On this 1,000-question test, coalition feedback scores +0.0147 F1 higher than supporting-passage labels (90% CI strictly above zero), exceeding the upper +0.02 equivalence threshold.