acceptodds
Under review as a conference paper at ICLR 2027

Learning to Select Evidence from Coalition Feedback

Abstract

Language models using retrieved passages or stored memories must select useful evidence under a context budget. A common utility measure, leave-one-out deletion, misses useful passages when another passage supplies the same fact. On an exact-copy stress test, its supporting-passage AUROC is 0.50, compared with 0.84 for a Banzhaf coalition teacher that measures contributions across different subsets. We use this coalition feedback to train CAMS, a 462,849-parameter setwise selector whose credit objective requires no supporting-passage labels and whose online selection makes no generative language-model calls. Across five pool settings, credit supervision yields answer F1 close to passage-label supervision in most evaluated budgets. On a 100-passage hard-negative pool at budget eight, the pure credit-trained selector improves answer F1 from 0.608 to 0.647 over dense retrieval. Its forward pass takes 1.2 ms on an H20, compared with 3.6 s for LLM reranking and 8.4 s for prompted set selection; measured embedding, feature, and forward components total 65 ms. Coalition feedback thus turns an offline measure of contextual usefulness into inexpensive evidence selection.

Primary area:
interpretability and explainable AI
Keywords:
memory-augmented language models, memory selection, counterfactual attribution, interaction-aware attribution, retrieval-augmented generation, passage selection, interpretability, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CYwWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies evidence selection for retrieval-augmented question answering with a limited context budget. It focuses on a limitation of leave-one-out attribution: when multiple passages provide the same information, removing one may not affect the answer, so useful passages can receive zero credit. The authors use Banzhaf values to estimate the contributions of passages across different subsets of retrieved evidence. They train CAMS, a small setwise selector operating on frozen embeddings, to predict these values without calling a language model during selection. Experiments examine evidence redundancy, downstream answer quality, and the effects of credit-based versus label-based supervision.

Critical strengths
  1. The paper demonstrates the limitation of LOO attribution under redundant evidence using a simple two-copy OR game and an exact-copy stress test. The downstream evaluation also reports paired confidence intervals for the performance improvements.

  2. The C/L/B comparison controls for architecture, input features, candidate pools, and random seeds. This allows the effects of different supervision targets to be compared under consistent experimental conditions.

  3. The statistical reporting includes paired bootstrap confidence intervals, standard deviations across seeds, and an explicit distinction between exploratory and confirmatory analyses.

Critical weaknesses
  1. The practical advantage of CAMS over simpler alternatives is limited. The main improvement is approximately 0.04 F1 on one candidate pool at one context budget. Meanwhile, a standard BGE cross-encoder, requiring 269 ms per query, substantially outperforms CAMS under tighter budgets. On Hard-100 with k=3, the cross-encoder achieves 0.665 F1 compared with 0.586 for CAMS. The paper does not establish when CAMS should be preferred over this baseline.

  2. The claim that credit supervision can replace labels is not supported by an appropriate statistical test. The conclusion relies on 21 of 22 comparisons showing no statistically significant difference, but non-significant differences do not establish equivalence. No equivalence margin is defined, and the only statistically significant comparison favors label supervision.

  3. The evaluation uses a single dataset with injected redundancy, and the improvement on the realistic candidate pool does not transfer to the second reader. For Llama, CAMS achieves 0.623 F1 compared with 0.627 for dense retrieval. The offline training requires 6.1 million reader calls and 68.8 GPU-hours, but the paper does not analyze whether this cost is justified by the online improvements compared with directly using a cross-encoder.

Questions for authors
  1. What motivated the choice of Banzhaf rather than Shapley values? Were both tested as supervision targets?

  2. The credit-versus-label claim would benefit from an equivalence test with a predefined margin. Can the authors provide this analysis?

  3. Given the 6.1 million offline reader calls, approximately how many online queries would be needed for CAMS to offset its training cost relative to the BGE cross-encoder?

2 comments
AuthorsOpenReview ↗

Response on Banzhaf vs Shapley, Feedback Statistics, and CE Cost

Thank you for your time and thorough review. We will respond to your comments point by point and detail the new experiments conducted. Addressing your questions on choosing Banzhaf over Shapley, the statistical basis for label substitution, cross-encoder quality and efficiency, and training costs, we performed a direct comparison with Shapley values, statistical equivalence testing between credit and labels, cost-quality analyses across new budgets, cross-encoder fusion experiments, and end-to-end timing measurements. Below, we explain the procedures, results, and corresponding paper conclusions for each point.

Equivalence Test Between Coalition Feedback and Passage Labels vs. Budget Comparisons

  1. Following Table 2 (where 21 of 22 comparisons were non-significant), we compared CAMS trained with coalition feedback (CAMS-C) against supporting-passage labels (CAMS-L) on 1,000 additional held-out questions on Hard-100 at k=8 using a two-sided ±0.02 Answer F1 equivalence bound.

  2. Coalition feedback achieved 0.6620 Answer F1 versus 0.6473 for passage labels, outperforming labels by +0.0147 F1 with a paired 90% confidence interval of [+0.0059, +0.0236]. The ±0.02 two-sided equivalence condition was not met because coalition feedback outperformed labels enough to push the upper confidence bound (+0.0236) above +0.02. Separately, in our E4 strict-target budget study, 37 of 44 non-deduplicated Qwen comparisons fell within an exploratory ±0.02 margin, which measures reduced budgets (128/256 vs. 1,024) rather than credit-vs.-label supervision.

  3. On this 1,000-question test, coalition feedback scores +0.0147 F1 higher than supporting-passage labels (90% CI strictly above zero), exceeding the upper +0.02 equivalence threshold.

AuthorsOpenReview ↗

Response on Banzhaf vs Shapley, Feedback Statistics, and CE Cost-2

Motivation for Banzhaf Values and Direct Student Comparison with Shapley Values

  1. We use Banzhaf values because they weight all passage subsets uniformly, matching uniform subset sampling, whereas Shapley weights depend on subset size and require different estimators. We directly trained and compared student selectors on Banzhaf versus Shapley targets across 20 settings under both matched sampled coalition counts and full enumeration.

  2. Under matched sampled coalition counts across the 20 settings, 19 paired difference confidence intervals crossed zero and 1 favored Banzhaf. Under full enumeration across the 20 settings, 3 confidence intervals favored Shapley and 17 crossed zero.

  3. Across 20 student-level settings, Banzhaf and Shapley targets yield statistically indistinguishable Answer F1 in 19 of 20 sampled settings (with 1 favoring Banzhaf) and 17 of 20 full-enumeration settings (with 3 favoring Shapley).

Comparison with Cross-Encoder (CE), Feature Fusion, and Speed-Quality Trade-off

  1. Building on the CE feature fusion in our manuscript, we evaluated CAMS with CE features under our newly matched Hard-100 k=8 setup, and timed full retrieval-to-answer execution across 40 questions each on HotpotQA pools, MuSiQue pools, and a single-H20 preloaded 32.5M-passage KILT setup with known-question candidate caching.

  2. On Hard-100 at k=8, pure CAMS scored 0.6486 Answer F1, CAMS+CE fusion scored 0.6955 (+0.0469 over pure CAMS), and full CE scored 0.7036 (0.0081 above fusion). In the 40-question end-to-end runs, CAMS had lower median latency and lower F1 than CE on HotpotQA (1.82s vs. 2.05s; F1 0.496 vs. 0.616, so CAMS is 0.23s faster and 0.120 F1 lower) and MuSiQue (2.174s vs. 2.306s; F1 0.2774 vs. 0.3268, 0.132s faster and 0.0494 F1 lower). On KILT, median latencies were 3.749s (Dense), 3.960s (CAMS), and 3.982s (CE).

  3. Adding CE features closes 85% of the F1 gap between pure CAMS and full CE on Hard-100 k=8, while pure CAMS provides a faster, lower-F1 operating point than full CE.

Offline Cost Arithmetic, New Generation Timing, and Cross-Dataset / Reader Scope

  1. We computed a conditional arithmetic ratio of historical offline cost (6.14M calls, 68.8 GPU-hours) to selection-stage latency savings vs. CE, timed fresh 1,500-question teacher generation (8,363,968 responses across 128, 256, and 1,024 budgets), and tested MuSiQue (250 pool and 250 KILT questions) and 512-token Llama-3.1-8B-Instruct.

  2. The arithmetic ratio equals ~2.76 million queries under the formal assumption of identical hardware, utilization, and quality (whereas CE has higher F1). For new teacher generation, we report only summed reader-call durations: 53.11, 106.12, and 423.85 hours at 128, 256, and 1,024 budgets. On MuSiQue at k=3, in-domain CAMS scored 0.3689 F1 (vs. 0.3114 Dense and 0.4488 CE), while zero-shot transfer (0.3260) and KILT retrieval (0.1414 vs. 0.1290 Dense) had CIs crossing zero. Retraining Llama at 512 tokens gave 22 deduplicated CIs crossing zero; running Llama on fixed E4 selections (26,818 safe / 28,076 strict responses) tested budget effects on another reader (Hard-100 k=8 strict 128−1024 diff −0.0061).

  3. Fresh teacher generation takes 53.11 and 106.12 summed call-hours at the smaller budgets, compared with 423.85 at 1,024. In-domain MuSiQue CAMS exceeds Dense by 0.0575 F1. The KILT and zero-shot CAMS-minus-Dense intervals include zero, while the Llama length-control rerun has 22 zero-containing intervals.

Thank you for helping us sharpen our statistical and practical analysis.

Reviewer_JcCzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses offline Banzhaf coalition feedback to train a lightweight evidence selector. It examines whether this feedback can overcome leave-one-out attribution’s sensitivity to redundant evidence and provide useful supervision without supporting-passage labels in the training loss.

Critical strengths
  1. The redundancy experiments clearly illustrate the limitations of leave-one-out attribution.
  2. Matched credit-only, label-only, and joint objectives help isolate the supervision source.
  3. Three training seeds, paired intervals, and matched deduplication controls strengthen the evaluation.
Critical weaknesses
  1. The benefit of coalition supervision over cheaper feedback remains unclear. The same selector is not trained on LOO targets, so the oracle attribution comparison does not isolate the downstream value of Banzhaf supervision.
  2. Generalization evidence is limited. The main matched evaluation covers five pool variants within HotpotQA, and Table 3 shows little or no gain over Dense on Llama for Natural-10 and Hard-100.
  3. End-to-end efficiency remains unmeasured. Section 5.2 reports component timings with some processing overhead excluded, rather than a complete CAMS-to-reader measurement.
Questions for authors
  1. How does the same selector perform when trained on LOO or lower-cost sampled coalition targets?
  2. Can the unified comparison be extended to independent datasets, with end-to-end timing of the trained pipeline?
1 comment
AuthorsOpenReview ↗

Response on Independent Data, Cheap Supervision, and Full Runtime

Thank you for your time and thorough review. We will address your comments point by point and outline the additional work conducted. Regarding your concerns about whether low-cost feedback can effectively train models, performance on an independent dataset, and end-to-end timing, we performed same-architecture leave-one-out comparisons, regenerated feedback under three budgets to train and evaluate student models, retrained on the MuSiQue dataset, and measured full-pipeline latency from retrieval to answering. Below, we detail the experimental procedures, results, and corresponding paper conclusions for each point.

Independent Dataset Evaluation (MuSiQue) and Reader Transfer (Llama)

  1. Beyond our unified five-pool HotpotQA evaluation, we tested MuSiQue via in-domain retraining (250 questions, 20 candidates, k=3) and 32.5M-passage KILT retrieval (250 questions), and tested Llama-3.1-8B-Instruct with a 512-token cap.

  2. On MuSiQue candidates, in-domain CAMS reached 0.3689 Answer F1, beating Dense (0.3114) and trailing CE (0.4488); zero-shot transfer scored 0.3260 (+0.0147 vs. Dense, 95% CI [−0.0246, +0.0540]). On KILT, Dense, CE, and CAMS scored 0.1290, 0.1671, and 0.1414 at k=3 (0.1609, 0.1852, 0.1517 at k=8), with CAMS–Dense CIs crossing zero. Retraining Llama at 512 tokens gave 22 deduplicated CIs vs. the 128-token control that all crossed zero. Running Llama on fixed E4 selections (26,818 safe and 28,076 strict contexts) tested how reduced teacher budgets transfer to another reader: Hard-100 k=8 strict F1 diff was −0.0061 [−0.0240, +0.0119] for 128−1024 and +0.0009 [−0.0135, +0.0160] for 256−1024.

  3. CAMS improves over Dense when trained in-domain on MuSiQue pools, while KILT retrieval, zero-shot transfer, and Llama retraining show no statistically significant gains over baseline controls.

End-to-End Retrieval-to-Answer Latency and Quality Trade-off

  1. To complement Section 5.2's stage-level timings, we timed full query-to-answer execution (retrieval, passage selection, and reader generation) across 40 questions per dataset on HotpotQA and MuSiQue pools, plus a 40-question single-H20 run over 32.5M KILT passages with preloaded models and known-question candidate caching.

  2. On HotpotQA pools, median end-to-end latencies for Dense, CAMS, and CE were 1.83s, 1.82s, and 2.05s, with same-run Answer F1 of 0.520, 0.496, and 0.616 (CAMS was 0.23s faster than CE while scoring 0.120 lower in F1). On MuSiQue pools, median latencies were 1.912s, 2.174s, and 2.306s, with F1 of 0.2556, 0.2774, and 0.3268 (CAMS was 0.132s faster than CE and 0.0494 lower in F1). On KILT, median latencies were 3.749s for Dense, 3.960s for CAMS, and 3.982s for CE.

  3. Across all three timing benchmarks, CAMS runs faster end-to-end than CE while CE achieves higher same-run Answer F1, so we report these direct speed-quality trade-offs.

Same-Architecture Selector Trained on LOO and Newly Generated Low-Budget Coalitions

  1. Extending our teacher-level ablations, we trained the identical CAMS student across five HotpotQA pools and three seeds on Leave-One-Out (LOO) targets, and generated 8,363,968 responses across 1,500 questions for 128, 256, and 1,024 coalition budgets to train 45 students each under two separate rules addressing 2,422 original extractor exceptions: safe (scoring exceptions as empty predictions) and strict (strict final-line rule).

  2. For Banzhaf vs. LOO students, Answer F1 was 0.6257 vs. 0.6112 (+0.0145) on Exact-copy k=3, 0.4283 vs. 0.4353 (−0.0070) on Exact-copy k=2, and 0.6486 vs. 0.6517 (−0.0031) on Hard-100 k=8. We report only summed reader-call durations: 53.11, 106.12, and 423.85 hours at 128, 256, and 1,024 budgets. Under non-deduplicated safe targets, Hard-100 k=8 kept similar scores across budgets (0.6569, 0.6565, 0.6564), whereas Natural-10 k=3 dropped at lower budgets (0.6590, 0.6602, 0.6738; 128−1024 diff −0.0148, 95% CI [−0.0282, −0.0028]).

  3. Banzhaf students beat LOO on Exact-copy k=3 while LOO is slightly higher on Exact-copy k=2 and Hard-100 k=8, and reducing budgets to 128/256 preserves F1 on Hard-100 k=8 but lowers F1 on Natural-10 k=3.

Thank you for your time and helpful feedback.

Reviewer_rVySOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates evidence selection with a restricted context budget in RAG. First, it is found that the leave-one-out (LOO) approach is unable to evaluate the impact of redundant evidence. Hence, in order to address this issue, Banzhaf values are employed to evaluate coalition contribution according to the Answer F1 metric of a frozen reader on various passage sets. These contributions are then utilized to develop a lightweight setwise selector, CAMS, that no longer requires generative LLM calls during online selection. Experiments are conducted on five different candidate-pool settings that are generated based on HotpotQA.

Critical strengths
  1. In the paper, the failure of LOO under redundant evidence is studied. As seen in Fig. 2(a), LOO gets a supporting-passage AUROC of 0.500 on Exact-copy, while Banzhaf obtains 0.837. Moreover, the oracle selection experiment demonstrates that for k=3, Banzhaf gets an Answer F1 of 0.796, which is significantly better than LOO's 0.555.
  2. The comparison of different types of supervision signals provides insight into their quality. Specifically, CAMS-C, CAMS-L, and CAMS-B have the same architecture and training procedure except for the objective, and as Table 2 shows, credit-only supervision gets Answer F1 very close to that of supporting-passage label supervision.
  3. In the paper, both the quality of selection and the computational costs are measured explicitly. In particular, Table 1 shows that CAMS-C significantly outperforms Dense and MMR on Hard-100 at k=8. Additionally, in Section 5.2, the selector forward time of 1.2 ms is separated from the embedding+feature+forward time of 65.24 ms, and the cost of offline feedback generation is reported.
Critical weaknesses
  1. The need for Banzhaf supervision is not adequately shown. Earlier work like CSM [1] trains a context selector based on LOO influence and captures interactions between contexts via global self-attention. SCARLet [2] learns the utility value of passages using context perturbations and leverages it to train the retriever. A key difference here is the use of Banzhaf credit, but there is no performance comparison with CAMS trained using LOO or other utility signals. Figure 2 shows that the Banzhaf teacher is better at attribution than LOO, but this does not mean that the advantage translates to the learned selector. In Section 5.1, the Spearman correlation between CAMS-C and the Banzhaf teacher is 0.269 compared with 0.251 for Dense Retrieval. It is still unclear how much of the performance gain is attributed to coalition supervision versus model structure and ranking training.
  2. The improvement in attribution does not sufficiently translate into the final evidence selection. Banzhaf computes the average contribution of a particular passage across subsets; however, CAMS conducts top-k selection based on scores, without directly optimizing the selected subset. According to Section 4.4, on Exact-copy at k=2, pure CAMS-C achieves 0.448 F1, with 89.7% of query selections containing duplicate sources. Deduplication improves F1 to 0.622, but the CAMS-L+dedup system, which uses label supervision, achieves F1=0.627. Section 5.1 reveals that even with exact Banzhaf scores, top-5 selection lags behind the oracle subset selection (0.836 vs. 0.868). This implies a significant discrepancy between Banzhaf-based attribution and the process of evidence selection, while the actual role of coalition supervision in solving the redundant evidence problem remains vague.
  3. It is unclear how much it costs to replace supporting-passage labels and what benefits can be achieved from such replacement. Unlike the training loss, where CAMS-C does not use passage labels, Appendix A reveals that checkpoint selection is still based on development support recall, and the deduplication thresholds depend on supporting annotations. Generating the training credits takes about 6.14 million reader calls and 68.8 GPU-hours. While the online forward latency is relatively low according to the paper, the latency of the full CAMS-to-reader pipeline and the impact of reducing coalition evaluations on final selection performance remain unclear.
  4. The evaluation scope is relatively narrow across datasets and readers. The five candidate-pool settings are all based on HotpotQA, and both Exact-copy and Paraphrase are synthetic redundancy settings. Table 3 reveals that after switching to the Llama reader, the performance gains of CAMS-C on Natural-10 and Hard-100 largely disappear, whereas the F1 gains with the Qwen reader are 0.033 and 0.039, respectively.
Questions for authors
  1. Could CAMS be trained with LOO credit or other perturbation-based utility signals under the same architecture and training setup, with a comparison of final Answer F1? Exact-copy and Hard-100 results would be especially interesting to see.
  2. Would it be possible to include a pointwise baseline, omitting the setwise encoder while leaving everything else intact, to test the contribution of setwise interaction learning?
  3. It is shown in Appendix F that even decreasing coalition evaluations from 1,024 to 128 gives a supporting-passage AUROC of 0.854. How would training CAMS with sampled Banzhaf targets affect Answer F1 and offline computation costs?
  4. How would CAMS-C perform if checkpoints and deduplication thresholds were selected without development supporting-passage labels?

[1] Deng et al. Influence Guided Context Selection for Effective Retrieval-Augmented Generation. NeurIPS, 2025. [2] Xu et al. Training a Utility-based Retriever Through Shared Context Attribution for Retrieval-Augmented Language Models. EMNLP, 2025.

3 comments
AuthorsOpenReview ↗

Response on Same-Architecture Supervision, Budget, Annotation Reliance, and Structure

Thank you for your time and careful review. We will address your questions point by point and describe the additional work undertaken. To investigate whether teacher attribution translates to student performance, structural contributions, duplicate context effects, low-cost feedback, label dependency, and cross-model generalization, we conducted same-architecture leave-one-out and size-matched passage-wise comparisons, deduplication diagnostics, retraining with feedback regenerated under three budgets, three validation model-selection strategies, and evaluations on MuSiQue and Llama. Below, we detail the procedures, results, and paper conclusions for each item.

Same-Architecture Comparison of Banzhaf vs. LOO Supervision and Related Work

  1. To test whether the teacher attribution gap in Figure 2a (0.837 Banzhaf vs. 0.500 LOO AUROC on Exact-copy) transfers to trained selectors, we trained the identical CAMS architecture (512-token cap, uniform regression weights) with Banzhaf vs. LOO targets across five pools and three seeds as an internal attribution ablation alongside our discussion of CSM and SCARLet.

  2. On Exact-copy at k=3, the Banzhaf student outperformed the LOO student by +0.0145 Answer F1 (0.6257 vs. 0.6112). Conversely, the LOO student scored +0.0070 higher on Exact-copy at k=2 (0.4353 vs. 0.4283) and +0.0031 higher on Hard-100 at k=8 (0.6517 vs. 0.6486).

  3. Banzhaf supervision provides a clear teacher-level attribution advantage under duplication and improves student F1 on Exact-copy k=3, whereas student F1 is slightly higher for LOO on Exact-copy k=2 and Hard-100 k=8.

Near-Parameter-Matched Pointwise vs. Setwise Encoder Ablation

  1. While Table 2 varied supervision signals within the setwise architecture, we tested the cross-passage encoder contribution by building a near-parameter-matched pointwise baseline (461,825 parameters vs. 462,849 for setwise, a 0.22% difference, with pool-level normalization and computation patterns still differing) across 20 non-deduplicated settings.

  2. Testing the interaction effect between setwise encoding and coalition supervision across these 20 settings yielded 18 confidence intervals that crossed zero and 2 confidence intervals below zero (favoring the pointwise control).

  3. The 20 settings provide no interval-supported positive interaction gain under this near-parameter-matched control. The interaction comparison directly measures whether setwise encoding adds benefit to the training target in these tested conditions.

Passage Attribution vs. Subset Selection under Redundancy

  1. Sections 4.4 and 5.1 showed that Banzhaf assigns equal positive credit to identical useful passages (unlike LOO's zero credit), so top-k ranking selects both copies unless deduplicated (Exact-copy k=2 pure CAMS-C gets 0.448 F1 with 89.7% duplicates vs. 0.622 deduplicated; top-5 exact Banzhaf gets 0.836 vs. 0.868 oracle subsets). We tested a Maximal Marginal Relevance (MMR)-style diversification rule against matched deduplication across 20 settings.

  2. Across the 20 settings, comparing MMR-style selection against our matched deduplication baseline produced 5 confidence intervals strictly below zero (where MMR lowered F1) and 15 intervals crossing zero, with zero settings improving over standard deduplication.

  3. The tested MMR-style rule lowered F1 in five settings with intervals below zero; the other 15 intervals included zero. We retain the matched similarity-deduplication method in this comparison.

Newly Generated Reduced Coalition Budgets (128 and 256) and Student Retraining

  1. To extend Appendix F from teacher AUROC to end-to-end students, we generated 8,363,968 responses across 1,500 questions at 128, 256, and 1,024 budgets. Because the original extractor had 2,422 exceptions, we report two separate sensitivity columns—safe (original extractor exceptions scored as empty predictions) and strict (strict final-line rule)—and retrained 45 students (5 pools × 3 budgets × 3 seeds) per rule.

  2. We report only summed generation call durations: 53.11, 106.12, and 423.85 hours at 128, 256, and 1,024 budgets (12.53% and 25.04% of 1,024). Under non-deduplicated safe targets, Hard-100 k=8 preserved near-identical F1 across budgets (0.6569, 0.6565, 0.6564), whereas Natural-10 k=3 dropped at lower budgets (0.6590, 0.6602, 0.6738; 128−1024 diff −0.0148, 95% CI [−0.0282, −0.0028]). Under non-deduplicated strict targets, 37 of 44 budget comparisons had 90% CIs within an exploratory ±0.02 margin.

  3. Cutting the coalition budget to 128 or 256 reduces generation call time by 75%–87% and maintains F1 on Hard-100 k=8, but incurs a −0.0148 F1 drop on Natural-10 k=3.

AuthorsOpenReview ↗

Response on Same-Architecture Supervision, Budget, Annotation Reliance, and Structure-2

Development Checkpoint Selection Without Supporting-Passage Labels

  1. Appendix A noted that while CAMS-C training loss uses no passage labels, development checkpoint selection and deduplication thresholds use supporting-passage annotations. We tested three label-free checkpoint selection rules on development pools: development Answer F1, teacher rank agreement, and minimizing student-teacher Mean Squared Error (MSE, evaluated across 18 settings under 0.95 deduplication using development QA answers for teacher targets).

  2. All three rules reduced test Answer F1 relative to support-recall selection. Specifically, teacher-MSE checkpoint selection failed by producing negative test F1 differences in 12 of 18 initial unadjusted 95% confidence intervals (one borderline interval crossed zero on a 20,000-resample check), including a −0.1363 Answer F1 drop on Natural-100 (w100) at k=3 (95% CI [−0.1861, −0.0859]).

  3. Teacher-MSE selection loses 0.1363 F1 on Natural-100/k=3. Among the tested development-selection rules, support-recall selection gives the stronger result for this setting. We therefore retain support labels for development selection in the reported pipeline.

Generalization to MuSiQue and Cross-Reader Runs with Llama

  1. We tested MuSiQue on 250 benchmark-pool questions (k=3, using support labels for checkpointing) and 250 KILT 32.5M-passage retrieval questions, and ran two 512-token Llama-3.1-8B-Instruct tests: retraining students on Llama credits, and running Llama on fixed E4 Qwen-student selections (26,818 safe and 28,076 strict responses; 1,309 and 1,358 hitting the 512-token cap).

  2. On MuSiQue pools at k=3, in-domain CAMS achieved 0.3689 F1, outperforming Dense (0.3114) and trailing CE (0.4488); zero-shot transfer reached 0.3260 (95% CI [−0.0246, +0.0540] vs. Dense) and KILT retrieval scored 0.1414 vs. 0.1290 Dense at k=3 (CI crossing zero). In 512-token Llama retraining, all 22 deduplicated CIs vs. the 128-token control crossed zero. Evaluating Llama on fixed E4 selections answered how teacher coalition budgets affect a different reader: on Hard-100 k=8 under strict targets, Llama F1 diff was −0.0061 [−0.0240, +0.0119] for 128−1024 and +0.0009 [−0.0135, +0.0160] for 256−1024.

  3. In-domain retraining on MuSiQue pools improves F1 over Dense by +0.0575, while KILT retrieval, zero-shot transfer, and 512-token Llama retraining show no statistically significant gains, and fixed-selection Llama runs show minimal difference across teacher budgets.

Thank you for your insightful questions and references.

Reviewer_rVySOpenReview ↗

Response to the Rebuttal

Thank you for the response. However, I do not think these additional results resolve my main concerns.

As reported in the response:

"On Exact-copy at k=3, the Banzhaf student outperformed the LOO student by +0.0145 Answer F1 (0.6257 vs. 0.6112). Conversely, the LOO student scored +0.0070 higher on Exact-copy at k=2 (0.4353 vs. 0.4283) and +0.0031 higher on Hard-100 at k=8 (0.6517 vs. 0.6486)."

These results suggest that although Banzhaf provides a better measure of evidence contribution, the current approach does not effectively translate this advantage into improved evidence selection.

I will therefore maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.