acceptodds
Under review as a conference paper at ICLR 2027

COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering

Abstract

Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily dependent on the choice of these training samples. In fact, we demonstrate that using more uncertain samples near the teacher's decision boundary can significantly deteriorate distillation performance. Motivated by this observation, we propose Counterfactual-guided Few-shot Distillation in open-book Question Answering (COFFEQA), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer. We provide geometric insights on how factual-counterfactual pairs can constrain the deviation between the teacher's and student's decision boundaries. We further introduce a new metric, Context Following Difficulty (CFD) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context. COFFEQA uses CFD-validated factual-counterfactual pairs for few-shot distillation. Experiments across five in-domain and three out-of-domain QA benchmarks using T5 and Qwen2.5 families show that COFFEQA outperforms factual-only distillation baselines under the same few-shot budget. For instance, on TriviaQA, COFFEQA improves over the factual-only baseline by average EM points and average F1 points under the T5 setting with 32 few-shot pairs.

Primary area:
interpretability and explainable AI
Keywords:
Counterfactuals, Few-shot Knowledge Distillation, Open-book QA

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MnZ8OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CoffeeQA, a few-shot knowledge distillation framework for open-book question answering. The paper first generates counterfactual contexts supporting alternative entity answers for factual samples, and then filters them using Counterfactual Feasibility Difficulty (CFD). This paper demonstrates that counterfactual pairing can benefit low-margin hard samples.

Critical strengths
  1. The paper discusses an important problem. Data quality and data selection are important to both knowledge distillation and LLM post-training, especially when the training budget is limited. Identifying a small set of informative examples could have substantial practical value.
  2. Using counterfactual examples to expose changes in the teacher’s behavior is intuitively meaningful.
Critical weaknesses
  1. The independent contribution of CoffeeQA is unclear. All reported results compare KD/LWD/SeqKD with “+CoffeeQA”. The paper does not separate the effects of CFD selection, counterfactual pairing, and additional supervision. Also, the average gains are modest.
  2. The experiments only compare CoffeeQA against baselines trained on a selected low-margin subset. It does not show that CoffeeQA is preferable to simpler data-selection strategies, such as random sampling or selecting medium-margin or high-margin factual examples. What if a simple sampling strategy match or outperform CoffeeQA without any counterfactual generation?
  3. The model family is outdated. The experiments only use T5 and Qwen2.5 in 2026. Since the paper propose a solution for efficient training and distillation, it should demonstrate that the method is useful on recent small-model families, such as Qwen3 or Llama 3.x. This is an essential experiment in the current era of rapid LLM development. The current experiments do not establish that CoffeeQA remains effective for contemporary models with stronger instruction-following and context-use abilities.
  4. The GPT-4o-mini counterfactual generation is not validated. The method directly asks GPT-4o-mini to rewrite contexts. No experiments or studies verify whether GPT-4o-mini reliably follows the generation instructions to produce valid and minimally modified counterfactual contexts.
Questions for authors
  1. Why is CoffeeQA evaluated only on open-book QA? Is the framework inherently limited to QA tasks, or is this only a limitation of the current experimental setup?
  2. How sensitive is CoffeeQA to the counterfactual generator? GPT-4o-mini may produce inconsistent or low-quality context rewrites. Have the authors compared it with a stronger model?
9 comments

We thank the reviewer for recognizing the contributions of our work and for providing constructive feedback and insightful questions. We address the reviewer's comments and questions below.

Part 1 (W1): Ablation Studies and Independent Contributions of COFFEQA

We had included several ablation studies in Appendix G of the original manuscript, examining the effects of CFD validation, CFD threshold selection. To further clarify the independent contributions of COFFEQA, we have now conducted additional experiments on T5, including component-wise CFD ablations, comparisons with simple entity replacement, alternative factual sampling strategies, and statistical significance analysis.

These existing and newly added experiments provide the following insights:

  • CFD validation and threshold selection: The original ablation in Appendix G.1 examines the effect of CFD verification and threshold selection. We further conducted a component-wise analysis comparing AA-only, CLR-only, and full CFD filtering. As shown in Table 1, full CFD achieves the highest average EM on four of the five datasets and improves overall average EM compared with either individual component, demonstrating the benefit of combining AA and CLR. These results are included in Appendix A.6.1 of the revised manuscript and discussed in our response to Reviewer 2, Part 4.
DatasetKD (paper)AA-only (highest AA)CLR-only (lowest CLR)Full CFD (paper COFFEQA)
NQ53.75 (−1.32)54.29 (-0.78)54.60 (-0.47)55.07
SQuAD75.37 (−1.93)77.15 (−0.15)77.14 (−0.16)77.30
HotpotQA54.79 (−6.04)60.29 (-0.54)60.88 (+0.05)60.83
NewsQA58.83 (−5.95)64.29 (−0.49)64.77 (−0.01)64.78
TriviaQA46.41 (−5.25)47.72 (−3.95)51.49 (−0.17)51.66

Table 1: Ablation study of CFD components. Exact Match (EM) scores for standard KD, AA-only, CLR-only, and full CFD filtering across five datasets, averaged over five few-shot budgets and five random seeds. Values in parentheses indicate EM differences relative to full CFD (COFFEQA). Bold indicates the best-performing method for each dataset.

  • Counterfactual generation: We compared LLM-based counterfactual rewriting with simple entity replacement while maintaining the same training settings. As shown in Table 2, LLM-based generation achieves higher average EM across all five datasets, with an overall average of 61.93 compared with 60.88 for entity replacement and 57.83 for standard KD. We also provide qualitative examples illustrating how LLM-based generation produces more coherent and internally consistent counterfactual contexts. These results are included in Appendix A.3 of the revised manuscript and discussed in our response to Reviewer 2, Part 2.
MethodNQSQuADNewsQATriviaQAHotpotQAAvg
KD (paper)53.754.5453.75_{4.54}75.371.8075.37_{1.80}58.836.1558.83_{6.15}46.412.7146.41_{2.71}54.796.9854.79_{6.98}57.8357.83
COFFEQA (paper)55.074.24\mathbf{55.07}_{4.24}77.300.99\mathbf{77.30}_{0.99}64.783.15\mathbf{64.78}_{3.15}51.663.78\mathbf{51.66}_{3.78}60.832.63\mathbf{60.83}_{2.63}61.93\mathbf{61.93}
COFFEQA (entity swap)53.405.1153.40_{5.11}76.931.0576.93_{1.05}63.573.5863.57_{3.58}50.305.6850.30_{5.68}60.222.8460.22_{2.84}60.8860.88

Table 2: Comparison of COFFEQA with simple entity replacement and standard KD. Factual Exact Match (EM) scores across five datasets using T5, averaged over different few-shot budgets. Standard deviations are shown as subscripts. Bold values indicate the best-performing method among the three approaches. The Avg column reports the mean across the five datasets.

  • Counterfactual pairing and factual selection: COFFEQA achieves higher overall average EM than factual-only KD across three additional factual sampling baselines: random, medium-margin, and high-margin sampling. These results demonstrate that counterfactual supervision provides benefits beyond the original factual selection strategy (see Appendix A.3 of the revised manuscript and response Part 2 table).

(continued in the next comment.)

(continuing ..)

  • Overall contribution and statistical significance: COFFEQA achieves a statistically significant improvement of 4.1 EM points (95% CI: [3.4, 4.8], p < 0.001) and 2.8 F1 points (95% CI: [2.5, 3.2], p < 0.001) over factual-only KD. Importantly, COFFEQA outperforms KD across all five random seeds and achieves higher EM in 24 of the 25 dataset and k-shot budget combinations, with no statistically significant decreases. (See Appendix A.2 of the revised manuscript. Also in Reviewer 1, Part 2 response).

These results provide additional evidence that counterfactual supervision contributes beyond the original factual selection strategy. We have incorporated the new ablations and clarified the contributions and limitations of each component in Appendix A.2, A.3, A.6 of the revised manuscript.

Part 2 (W2): Effect of Factual Sampling Baselines

COFFEQA provides additional performance gains across alternative factual sampling baselines. To investigate whether the improvements of COFFEQA arise from counterfactual supervision or factual sample selection, we conducted additional experiments on T5 using three factual sampling baselines: random, medium-margin, and large-margin sampling. We compared factual-only KD against COFFEQA under each baseline across five datasets and five few-shot budgets, using five random seeds. As shown in Table 3, COFFEQA improves average EM over factual-only KD across all five datasets under random and large-margin sampling. Under the medium-margin baseline, COFFEQA improves performance on four of the five datasets.

MethodNQSQuADNewsQATriviaQAHotpotQAAvg
KD (paper)53.754.5453.75_{4.54}75.371.8075.37_{1.80}58.836.1558.83_{6.15}46.412.7146.41_{2.71}54.796.9854.79_{6.98}57.8357.83
COFFEQA (paper)55.074.24\mathbf{55.07}_{4.24}77.300.99\mathbf{77.30}_{0.99}64.783.15\mathbf{64.78}_{3.15}51.663.78\mathbf{51.66}_{3.78}60.832.63\mathbf{60.83}_{2.63}61.93\mathbf{61.93}
Random KD54.933.3654.93_{3.36}76.492.2176.49_{2.21}57.776.1357.77_{6.13}48.635.2048.63_{5.20}59.762.6459.76_{2.64}59.5259.52
Random COFFEQA56.891.56\mathbf{56.89}_{1.56}77.780.86\mathbf{77.78}_{0.86}63.613.36\mathbf{63.61}_{3.36}51.514.32\mathbf{51.51}_{4.32}65.941.00\mathbf{65.94}_{1.00}63.15\mathbf{63.15}
Medium margin KD60.301.5260.30_{1.52}78.180.2578.18_{0.25}71.543.4871.54_{3.48}53.943.4453.94_{3.44}73.111.80\mathbf{73.11}_{1.80}67.4167.41
Medium margin COFFEQA60.731.16\mathbf{60.73}_{1.16}78.190.50\mathbf{78.19}_{0.50}72.562.74\mathbf{72.56}_{2.74}55.950.93\mathbf{55.95}_{0.93}72.202.0172.20_{2.01}67.93\mathbf{67.93}
Large-margin KD54.021.5454.02_{1.54}77.941.0677.94_{1.06}65.231.2265.23_{1.22}51.702.7251.70_{2.72}66.151.1766.15_{1.17}63.0163.01
Large-margin COFFEQA56.302.36\mathbf{56.30}_{2.36}78.060.56\mathbf{78.06}_{0.56}66.231.54\mathbf{66.23}_{1.54}55.591.90\mathbf{55.59}_{1.90}66.640.84\mathbf{66.64}_{0.84}64.56\mathbf{64.56}

Table 3: Comparison of factual sampling baselines for KD and COFFEQA. In-domain Exact Match (EM) scores are reported across five datasets, averaged over different few-shot budgets in the T5 setting. The baselines include low-margin, random, medium-margin, and high-margin sampling strategies. Standard deviations are shown as subscripts. Bold values indicate the better-performing method within each KD–COFFEQA pair. The Avg column reports the mean across the five datasets.

We also observe that the medium-margin factual-only KD baseline outperforms the original low-margin COFFEQA configuration on all five datasets, highlighting the importance of factual sample selection. Nevertheless, the improvements observed under matched factual sampling baselines demonstrate that counterfactual supervision provides additional benefits beyond factual selection alone. We have included these comparisons in Appendix A.3 of the revised manuscript.

(continued in the next comment.)

(continuing ..)

Part 3 (W3): Generalization of COFFEQA to Recent Model Families (Qwen3)

COFFEQA consistently improves distillation performance with the newer Qwen3 model family. To evaluate whether COFFEQA remains effective with a more recent model family, we extended our experiments to Qwen3, using Qwen3-8B as the teacher and Qwen3-1.7B as the student. Both KD and COFFEQA use the same experimental settings, with few-shot budgets ranging from 16 to 256 across five random seeds. As shown in Table 4, COFFEQA achieves higher average EM than KD across all five datasets. The largest improvements are observed on TriviaQA (42.91 to 52.80), HotpotQA (54.37 to 62.88), and NewsQA (40.83 to 47.35), with additional gains on NQ (61.11 to 63.83) and SQuAD (62.44 to 66.29). These results demonstrate that the benefits of counterfactual supervision extend to newer model families, further supporting the generalizability of COFFEQA. We have included these results in Appendix A.4 of the revised manuscript.

DatasetMethod163264128256Avg
NQKD62.060.21\mathbf{62.06}_{0.21}62.960.7062.96_{0.70}62.550.8662.55_{0.86}60.672.2060.67_{2.20}57.291.8757.29_{1.87}61.112.3161.11_{2.31}
COFFEQA60.390.3060.39_{0.30}64.031.18\mathbf{64.03}_{1.18}64.970.97\mathbf{64.97}_{0.97}65.362.71\mathbf{65.36}_{2.71}64.381.85\mathbf{64.38}_{1.85}63.831.99\mathbf{63.83}_{1.99}
SQuADKD54.330.6554.33_{0.65}63.081.5863.08_{1.58}66.630.8966.63_{0.89}65.450.3065.45_{0.30}62.700.8562.70_{0.85}62.444.8162.44_{4.81}
COFFEQA65.331.24\mathbf{65.33}_{1.24}68.511.35\mathbf{68.51}_{1.35}67.060.66\mathbf{67.06}_{0.66}66.211.06\mathbf{66.21}_{1.06}64.340.84\mathbf{64.34}_{0.84}66.291.60\mathbf{66.29}_{1.60}
HotpotQAKD54.082.2954.08_{2.29}43.264.1443.26_{4.14}56.854.9956.85_{4.99}59.181.7459.18_{1.74}58.501.4158.50_{1.41}54.376.5054.37_{6.50}
COFFEQA64.082.47\mathbf{64.08}_{2.47}61.184.79\mathbf{61.18}_{4.79}62.831.31\mathbf{62.83}_{1.31}63.631.09\mathbf{63.63}_{1.09}62.701.73\mathbf{62.70}_{1.73}62.881.10\mathbf{62.88}_{1.10}
NewsQAKD36.441.3036.44_{1.30}37.313.5537.31_{3.55}45.212.6345.21_{2.63}42.291.2742.29_{1.27}42.921.9242.92_{1.92}40.833.7840.83_{3.78}
COFFEQA40.384.33\mathbf{40.38}_{4.33}49.030.97\mathbf{49.03}_{0.97}49.931.42\mathbf{49.93}_{1.42}50.221.80\mathbf{50.22}_{1.80}47.182.27\mathbf{47.18}_{2.27}47.354.07\mathbf{47.35}_{4.07}
TriviaQAKD28.393.4428.39_{3.44}39.411.6639.41_{1.66}47.474.1847.47_{4.18}50.331.4150.33_{1.41}48.950.8648.95_{0.86}42.919.1742.91_{9.17}
COFFEQA51.284.21\mathbf{51.28}_{4.21}47.831.18\mathbf{47.83}_{1.18}53.321.18\mathbf{53.32}_{1.18}54.741.54\mathbf{54.74}_{1.54}56.841.36\mathbf{56.84}_{1.36}52.803.43\mathbf{52.80}_{3.43}

Table 4: Qwen3 In-Domain Exact Match (EM) Comparison of KD and COFFEQA. Results are reported across five datasets and five few-shot budgets (k = 16–256), using Qwen3-8B as the teacher and Qwen3-1.7B as the student. Values represent mean EM over five random seeds, with standard deviations shown as subscripts. Bold values indicate the better-performing method.

Part 4 (Q1): Generalizability Beyond Open-Book QA

COFFEQA is not inherently limited to open-book QA, but our current formulation focuses on this setting because counterfactuals can be generated by modifying the supporting context while keeping the question fixed. We also focus on named-entity answers, which makes it easier to construct and validate alternative answers.

Extending this approach to other generative tasks is challenging because small input changes may affect multiple parts of the output, and there may not be a single correct alternative response. Nevertheless, the framework could be adapted to multiple-choice QA by modifying the evidence to support a different answer option, mathematical reasoning by changing numerical values or premises and updating the solution, or summarization by modifying key facts and their corresponding summaries. Such extensions would require task-specific counterfactual generation and validation methods. We have clarified in Appendix B of the revised manuscript that this is a limitation of our current experimental setup rather than the general idea of counterfactual-guided distillation.

(continued in the next comment.)

Reviewer_upXNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes COFFEQA, a counterfactual-guided few-shot knowledge distillation (KD) framework for generative open-book question answering. The work starts from the observation that factual samples near the teacher’s decision boundary (i.e., those with small prediction margins) degrade few-shot distillation performance. Building on this, the authors use GPT-4o-mini to generate minimally perturbed counterfactual context–answer pairs around these hard samples, providing additional supervision near the boundary. To filter out low-quality synthetic data, they introduce Context Following Difficulty (CFD), a validation metric that combines Answer Agreement (AA) and Contextual Loss Ratio (CLR). They also present a geometric analysis that bounds the Hausdorff distance between teacher and student context–response boundaries (Proposition 1). Experiments with T5 and Qwen2.5 on five in-domain and three out-of-domain QA benchmarks show that COFFEQA consistently outperforms standard KD, SeqKD, and Layer-Wise Distillation (LWD) baselines.

Critical strengths
  1. Practical problem with broad empirical evaluation. Data-efficient KD for generative QA under tight few-shot budgets is a relevant and practical problem, and systematically exploiting data near the decision boundary in a generative setting has practical value. The authors evaluate both encoder-decoder (T5) and decoder-only (Qwen2.5) architectures and report both in-domain and out-of-domain transfer. This breadth is the main reason the empirical results are worth taking seriously.

  2. Well-motivated validation metric (CFD). CFD measures both target-answer alignment (AA) and contextual reliance (CLR) when filtering synthetic counterfactuals, rather than relying on answer agreement alone. In particular, it penalizes counterfactual contexts that reduce teacher loss without actually guiding the target generation. This is an intuitive and potentially reusable heuristic for synthetic data filtering.

Critical weaknesses
  1. Confounded baseline comparison and missing control experiments. The central premise is that factual samples with small teacher margins degrade distillation. However, the factual-only baseline is built only from the k samples with the smallest margins. COFFEQA is therefore compared against a deliberately handicapped baseline rather than a well-configured factual KD baseline. Under the same budget k, the following controls are needed:
  • Random Factual-only (k) vs. COFFEQA (k/2 Random Factual + k/2 Counterfactual). This would show that the gain is not simply an artifact of escaping poor-margin factual selection.
  • Large-Margin Factual-only (k) vs. COFFEQA (k/2 Large-Margin Factual + k/2 Counterfactual). This would show whether counterfactual supervision actually outperforms high-confidence factual supervision.
  • Simple Entity Replacement (k/2 Factual + k/2 heuristically entity-swapped samples) vs. COFFEQA. This would disentangle whether LLM-based context rewriting teaches genuine contextual dependence, or whether the student simply fits local entity-substitution patterns. This is consequential because, without these controls, the main empirical claim cannot be attributed to counterfactual supervision itself.
  1. Theoretical issues in Proposition 1.
  • Limited novelty over prior work. Few-shot distillation with counterfactual pairs, and bounding the Hausdorff distance by perturbation size and boundary coverage (≤α+ϵ\le \alpha + \epsilon), already appear in classification-based counterfactual distillation (e.g., CoD, Hamman et al., 2025). It is unclear what is mathematically new in the extension to generative QA.
  • Ill-defined response boundary in generative space. In binary classification, P(A)=P(A′)=0.5P(A) = P(A') = 0.5 rigorously defines a decision boundary. In generative QA, however, Pt(A∣Q,c)=Pt(A′∣Q,c)P_t(A \mid Q, c) = P_t(A' \mid Q, c) does not guarantee a transition boundary between AA and A′A', because a third, unrelated response A′′A'' may dominate both (Pt(A′′∣Q,c)>Pt(A∣Q,c)=Pt(A′∣Q,c)P_t(A'' \mid Q, c) > P_t(A \mid Q, c) = P_t(A' \mid Q, c)). The authors state that CFD filters out such A′′A'', but CFD is an empirical heuristic and cannot serve as a mathematical guarantee within Proposition 1.
  • Circular dependency and curse of dimensionality in Assumption (3). Assumption (3), ϵ\epsilon-spread boundary coverage, requires the selected training set to cover not only the teacher boundary MtM_t but also the student boundary MsM_s within ϵ\epsilon. Yet MsM_s is determined only after distillation. Requiring the training set to cover a future student boundary introduces a circular dependency. Moreover, in a high-dimensional context-embedding space, it is implausible that only 16–256 few-shot points form an ϵ\epsilon-net over the entire boundary.

This is consequential because Proposition 1 is presented as the theoretical justification for the method, and these issues substantially weaken that justification.

Questions for authors
  1. What are the results when COFFEQA is compared against Random Factual-only (k) and Large-Margin Factual-only (k) baselines under the same budget? Does the same improvement trend hold?

  2. Does a naive string/entity replacement baseline, without LLM-based context rewriting, achieve comparable distillation gains?

  3. How do the authors theoretically justify Assumption (3), which conditions the training set on the student boundary MsM_s that is unknown before training?

  4. Could the authors provide component-wise ablations of CFD (AA-only filtering and CLR-only filtering), in addition to the current with/without CFD comparison? In Eq. (3), when CLR → 0, CFD can remain small (i.e., “good”) even if AA is very low, so a low CFD score does not guarantee the intended answer. In addition, a low CLR indicates reduced loss given the context, but it does not rule out shortcut learning or answer leakage.

  5. The generated data naturally form pairwise preferences: (Q,C,A)≻(Q,C,A′)(Q, C, A) \succ (Q, C, A') and (Q,C′,A′)≻(Q,C′,A)(Q, C', A') \succ (Q, C', A). Have the authors compared KD/SeqKD against DPO or contrastive distillation using the same counterfactual pairs under the same data budget?

  6. The teachers evaluated (T5-Large 770M, Qwen2.5-7B) are relatively small. It would be informative to verify whether the boundary-margin effect and the gains persist with larger teachers (e.g., 70B-scale).

  7. The three-stage counterfactual generation with GPT-4o-mini, plus CFD validation through the teacher (two forward-loss passes and one generation pass), adds non-trivial computational and API cost. A cost-benefit analysis against stronger standard distillation under an equal compute budget (e.g., Rubric-based KD, CoT distillation) would strengthen the paper.

8 comments

We thank the reviewer for recognizing the contributions of our work and for providing constructive feedback and insightful questions. We address the reviewer's comments and questions below.

Part 1 (W1 & Q1): Effect of Factual Sampling Baselines

COFFEQA generally outperforms factual sampling baselines. We evaluated the following additional factual sampling baselines: randomly selecting factual examples and selecting high-margin factual examples (large-margin). As shown in table 1, COFFEQA generally outperforms KD under both strategies. For random sampling, COFFEQA achieves large improvements on NewsQA and HotpotQA. These results suggest that counterfactual supervision can be beneficial even when the factual examples are not selected using the lowest teacher margins. We have included this comparison in Appendix A.3 of the revised manuscript.

MethodNQSQuADNewsQATriviaQAHotpotQAAvg
KD (paper)53.754.5453.75_{4.54}75.371.8075.37_{1.80}58.836.1558.83_{6.15}46.412.7146.41_{2.71}54.796.9854.79_{6.98}57.8357.83
COFFEQA (paper)55.074.24\mathbf{55.07}_{4.24}77.300.99\mathbf{77.30}_{0.99}64.783.15\mathbf{64.78}_{3.15}51.663.78\mathbf{51.66}_{3.78}60.832.63\mathbf{60.83}_{2.63}61.93\mathbf{61.93}
Random KD54.933.3654.93_{3.36}76.492.2176.49_{2.21}57.776.1357.77_{6.13}48.635.2048.63_{5.20}59.762.6459.76_{2.64}59.5259.52
Random COFFEQA56.891.56\mathbf{56.89}_{1.56}77.780.86\mathbf{77.78}_{0.86}63.613.36\mathbf{63.61}_{3.36}51.514.32\mathbf{51.51}_{4.32}65.941.00\mathbf{65.94}_{1.00}63.15\mathbf{63.15}
Medium margin KD60.301.5260.30_{1.52}78.180.2578.18_{0.25}71.543.4871.54_{3.48}53.943.4453.94_{3.44}73.111.80\mathbf{73.11}_{1.80}67.4167.41
Medium margin COFFEQA60.731.16\mathbf{60.73}_{1.16}78.190.50\mathbf{78.19}_{0.50}72.562.74\mathbf{72.56}_{2.74}55.950.93\mathbf{55.95}_{0.93}72.202.0172.20_{2.01}67.93\mathbf{67.93}
Large-margin KD54.021.5454.02_{1.54}77.941.0677.94_{1.06}65.231.2265.23_{1.22}51.702.7251.70_{2.72}66.151.1766.15_{1.17}63.0163.01
Large-margin COFFEQA56.302.36\mathbf{56.30}_{2.36}78.060.56\mathbf{78.06}_{0.56}66.231.54\mathbf{66.23}_{1.54}55.591.90\mathbf{55.59}_{1.90}66.640.84\mathbf{66.64}_{0.84}64.56\mathbf{64.56}

Table 1: Comparison of factual sampling baselines for KD and COFFEQA. In-domain Exact Match (EM) scores are reported across five datasets, averaged over different few-shot budgets in the T5 setting. The baselines include low-margin, random, medium-margin, and high-margin sampling strategies. Standard deviations are shown as subscripts. Bold values indicate the better-performing method within each KD–COFFEQA pair. The Avg column reports the mean across the five datasets.

Part 2 (W1 & Q1): Comparison with simple entity-replacement baseline

COFFEQA outperforms simple entity-replacement baseline. We evaluated a simple entity-replacement baseline, where the original answer is replaced with a randomly selected training-set entity of the same type. For a fair comparison, all methods use the same factual examples, training budgets, and five random seeds. As shown in Table 2, COFFEQA's LLM-based counterfactual generation outperforms simple entity replacement across all five datasets on average, achieving an overall EM of 61.93, compared with 60.88 for entity replacement and 57.83 for factual-only KD. Both counterfactual approaches generally outperform KD, demonstrating the benefits of counterfactual supervision, while the additional gains from COFFEQA highlight the importance of its generation strategy. We have included this comparison in Appendix A.3 of the revised manuscript.

MethodNQSQuADNewsQATriviaQAHotpotQAAvg
KD (paper)53.754.5453.75_{4.54}75.371.8075.37_{1.80}58.836.1558.83_{6.15}46.412.7146.41_{2.71}54.796.9854.79_{6.98}57.8357.83
COFFEQA (paper)55.074.24\mathbf{55.07}_{4.24}77.300.99\mathbf{77.30}_{0.99}64.783.15\mathbf{64.78}_{3.15}51.663.78\mathbf{51.66}_{3.78}60.832.63\mathbf{60.83}_{2.63}61.93\mathbf{61.93}
COFFEQA (entity swap)53.405.1153.40_{5.11}76.931.0576.93_{1.05}63.573.5863.57_{3.58}50.305.6850.30_{5.68}60.222.8460.22_{2.84}60.8860.88

Table 2: Comparison of COFFEQA with simple entity replacement and standard KD. Factual Exact Match (EM) scores across five datasets using T5, averaged over different few-shot budgets. Standard deviations are shown as subscripts. Bold values indicate the best-performing method among the three approaches. The Avg column reports the mean across the five datasets.

(continued in the next comment.)

(continuing ..)

LLM-based counterfactual generation produces more plausible and internally consistent contexts than simple entity replacement. As shown in Table 3, simple entity replacement can introduce implausible or internally inconsistent information, such as "6 million years" as an age or "50 billion shillings or 240 pence" as a currency conversion. In contrast, LLM-based generation produces more plausible and internally consistent examples, such as "30 years" as an age, "Saturday" as a day, and "22 shillings, or 264 pence" as a currency conversion. LLM-based rewriting can also update related references that simple string replacement may miss. Overall, the qualitative examples in Table 3 highlight the potential benefits of LLM-based rewriting over simple entity replacement for generating plausible and consistent counterfactual contexts.

Dataset: question (original answer)Random same-type entityCOFFEQA (GPT-4o-mini)
NQ: How many shillings in a pound? (20)“12 pence in a shilling and 50 billion shillings, or 240 pence, in a pound”“… 22 shillings, or 264 pence, in a pound”
NQ: Minimum age to become Prime Minister of India? (25)“be above 6 million years of age”“be above 30 years of age”
NewsQA: When was he found guilty? (Tuesday)“Abdulla was found guilty 1920 of conspiracy to murder”“Abdulla was found guilty Saturday of conspiracy to murder”
HotpotQA: In what city is Beautiful Indonesia Miniature Park? (Jakarta)“located in Taman Mini Indonesia Indah (TMII), Nepal, Indonesia”“located in Taman Mini Indonesia Indah (TMII), Bekasi, Indonesia”
SQuAD: Poorest US city as of 2004? (Detroit)“behind only Bosnia and Herzegovina, Michigan (ranked #1)”“behind only Grand Rapids, Michigan (ranked #1)”

Table 3: Qualitative comparison of counterfactual generation strategies. Representative examples generated using simple entity replacement and COFFEQA (GPT-4o-mini).

Part 3 (W2 & Q2): Extension of Proposition 1 to Generative QA and Its Assumptions

Extending the theoretical analysis to generative QA. Unlike binary classification, generative QA involves an open-ended, context-dependent answer space, making decision-boundary analysis more challenging. In particular, a third answer (A'') may dominate between factual and counterfactual contexts, complicating the transition between (A) and (A'). We acknowledge that the Hausdorff bound and its geometric argument are adapted from classification-based counterfactual distillation by Hamman et al. (2025) [1]. Our contribution extends this analysis to open-book generative QA by formulating context–response boundaries with the question fixed and constructing factual-counterfactual pairs through minimal context edits. Specifically, we define the (A/A') boundary where both answers tie for the highest probability:

MA,A′=c∈C:p(A∣Q,c)=p(A′∣Q,c)≥max⁡A′′∉A,A′p(A′′∣Q,c)\mathcal{M}_{A,A'} = \\{c \in \mathcal{C} : p(A \mid Q,c) = p(A' \mid Q,c) \geq \max_{A'' \notin \\{A,A'\\}} p(A'' \mid Q,c)\\}

This formulation explicitly accounts for competing responses in the generative setting and provides a geometric framework for analyzing factual-counterfactual transitions.

(continued in the next comment.)

(continuing ..)

Boundary coverage (A3). Although the student's boundary is unknown before distillation and few-shot samples cannot densely cover an arbitrary high-dimensional space, two properties motivate meaningful boundary coverage under limited supervision:

  • Lower-dimensional manifold of language representations. Previous studies suggest that natural-language representations exhibit lower-dimensional manifold structure despite being embedded in high-dimensional spaces [2, 3]. Consequently, boundary alignment is primarily relevant on the data manifold, where meaningful inputs occur, rather than throughout the entire representation space. This reduces the effective complexity of the regions requiring coverage.

  • Smooth decision boundaries and knowledge distillation. Prior studies suggest that knowledge distillation transfers decision-boundary information [4, 5] and exhibits statistical and regularization properties [6–8]. These findings motivate the expectation that teacher and student boundaries may remain smoothly aligned even between observed training regions. In contrast, a highly oscillating student boundary could deviate substantially in regions without direct supervision. Figure 6 in Appendix A.1 illustrates these scenarios, highlighting how smooth alignment can make meaningful boundary coverage plausible with fewer training examples.

Together, these considerations suggest that fewer samples may provide meaningful coverage on the lower-dimensional data manifold, where teacher-student boundary alignment is most relevant. They provide geometric motivation for A3 under few-shot supervision while recognizing its role as an assumption in the theoretical analysis.

What CFD and exact agreement provide. CFD empirically evaluates counterfactual endpoints through Answer Agreement (AA), which assesses whether the teacher generates the intended alternative answer, and Contextual Loss Ratio (CLR), which measures whether that answer becomes more predictable given the edited context. All accepted counterfactuals satisfy AA ≥ 0.4, and 88–96% achieve AA = 1 across datasets. These findings provide empirical support for the intended context-dependent behavior and complement our geometric formulation. While CFD evaluates the endpoints rather than the entire edit path, it provides a practical mechanism for selecting meaningful counterfactual pairs and motivates further investigation of multi-answer boundary interactions.

Revisions and future directions. We have clarified the connection to Hamman et al. [1], refined the context–response boundary formulation, and strengthened the discussion of the assumptions underlying Proposition 1. These revisions are incorporated into Section 2.2 and Appendix A.1 of the revised manuscript. Building on this framework, we are interested in extending the analysis to sequence-level margins, intermediate contexts along edit paths, multiple counterfactuals for broader boundary coverage, and semantically equivalent answer forms.

[1] Hamman et al. (2026). Few-shot knowledge distillation of LLMs with counterfactual explanations. NeurIPS.

[2] Lee et al. (2025). Shared global and local geometry of language model embeddings. COLM.

[3] Tulchinskii et al. (2023). Intrinsic dimension estimation for robust detection of AI-generated texts. NeurIPS.

[4] Heo et al. (2019). Knowledge transfer via distillation of activation boundaries formed by hidden neurons. AAAI.

[5] Ojha et al. (2023). What knowledge gets distilled in knowledge distillation? NeurIPS.

[6] Menon et al. (2021). A statistical perspective on distillation. ICML.

[7] Zhou et al. (2021). Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. ICLR.

[8] Mobahi et al. (2020). Self-distillation amplifies regularization in Hilbert space. NeurIPS.

(continued in the next comment.)

Reviewer_iRtMOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes COFFEQA, a method for few-shot distillation in open-book question answering. It pairs factual samples with counterfactuals whose modified contexts support alternative answers. Context Following Difficulty (CFD) combines Answer Agreement and Contextual Loss Ratio to validate these counterfactuals before distillation. The paper also gives a geometric analysis of teacher–student boundary alignment. Experiments cover five in-domain and three out-of-domain QA benchmarks with T5 and Qwen2.5 models, using KD, SeqKD, and LWD. Results show gains in overall average performance and in many individual settings, although some settings show decreases. The total student training budget remains fixed at k instances.

Critical strengths
  1. A useful approach to limited-data supervision. Pairing factual samples with counterfactuals gives the student direct examples of how answers should change with contextual evidence. CFD adds a practical check that the teacher follows the intended answer and benefits from the modified context. This is a meaningful contribution to few-shot QA distillation.

  2. Broad empirical evaluation. The paper studies two model families, several distillation methods, and both in-domain and out-of-domain benchmarks. It also evaluates modified contextual evidence and includes ablations of CFD and its threshold. Together, these experiments support the value of the approach beyond a single benchmark.

Critical weaknesses
  1. The theoretical result has limited practical support. Proposition 1 assumes exact teacher–student agreement on paired samples and sufficient coverage of both boundaries. These are strong conditions in a few-shot setting, and the experiments do not establish that they hold. The result offers geometric intuition, but its connection to the measured gains remains unclear.

  2. The cost and stability of the gains need clearer evidence. COFFEQA does not increase the number of student training instances, but it adds counterfactual generation, failed-candidate retries, and CFD teacher evaluations. The cost analysis should clearly state whether all these steps are included. The paper reports means and standard deviations over five seeds, but no paired comparisons, confidence intervals, or significance tests. This makes some small gains difficult to assess, particularly given the decreases in several settings.

Questions for authors
  1. Does the reported cost include generation, all failed-candidate retries, and CFD teacher evaluations? What is the cost of each step?
  2. Can you provide paired seed comparisons or confidence intervals for the smaller gains? Which validation data were used to select the CFD threshold?
  3. How are the context embeddings, distance metric, and boundary coverage in Proposition 1 related to the implemented method? Is there empirical evidence for the assumed coverage?
6 comments

We thank the reviewer for recognizing the contributions of our work and for providing constructive feedback and insightful questions. We address the reviewer's comments and questions below.

Part 1 (W1 & Q3): Empirical Assessment of Theoretical Assumptions and Practical Relevance

We expand our discussion of the three assumptions underlying Proposition 1 through empirical analyses and geometric intuition. For Assumption A1, we examine the magnitude of counterfactual perturbations in textual and embedding spaces. For Assumption A2, we evaluate teacher-student agreement empirically across different few-shot budgets. For Assumption A3, we provide geometric motivation based on the lower-dimensional manifold of language representations and the smooth boundary alignment encouraged by knowledge distillation. These analyses clarify the assumptions and their relevance to the proposed method.

A1 (Minimal Perturbations): To verify that our generated counterfactuals satisfy the small perturbation assumption, we empirically measure their differences from factual samples in both textual and embedding spaces. As shown in Table 1, accepted counterfactuals modify a median of 1–5 words, corresponding to only 0.7–1.4% of the original context, while approximately preserving context length. We measure embedding perturbations using the L2 distance between mean-pooled T5-Large teacher encoder representations of factual and counterfactual inputs. Low-margin examples exhibit smaller perturbations than randomly selected examples on four of the five datasets, with comparable distances on TriviaQA. Furthermore, teacher margin and perturbation size show a statistically significant positive correlation on these four datasets (p < 0.001). These findings provide empirical support for Assumption A1.

DatasetTextual perturbationEmbedding perturbation
Median words changedMedian share changedLength ratio C′/CMedian perturbation distance (low-margin / random)Correlation between teacher margin and perturbation size
NQ11.4%1.000.120 / 0.1830.45 (p < 0.001)
SQuAD11.1%1.000.118 / 0.1280.30 (p < 0.001)
NewsQA30.7%0.990.178 / 0.2260.24 (p < 0.001)
TriviaQA50.7%1.000.085 / 0.085−0.03 (n.s.)
HotpotQA21.0%1.000.111 / 0.2000.18 (p < 0.001)

Table 1: Empirical evaluation of Assumption A1: Counterfactual perturbation size. Textual perturbations are measured by the number of modified words and context length ratios. Embedding perturbations are measured using L2 distances between mean-pooled T5-Large encoder representations of factual and counterfactual inputs, comparing low-margin and randomly selected examples. Correlations are measured using Spearman's ρ between teacher margin and perturbation size, with corresponding significance levels.

(continued in the next comment.)

(continuing ..)

A2 (Exact Answer Generation): We empirically evaluated how often Assumption A2 holds in practice. As shown in Table 2, exact teacher–student agreement on both factual and counterfactual samples increases from 60.0% at k = 16 to 83.1% at k = 128, remaining at 83.0% for k = 256. When the teacher correctly answers both samples, agreement is substantially higher (93–100%), suggesting that many disagreements occur when the teacher makes mistakes. We observe a similar pattern across all five datasets, with conditional agreement ranging from 84% to 95%. Overall, these results suggest that A2 is approximately satisfied in practice, particularly at larger training budgets, although exact agreement is not achieved for every pair.

Metrick = 16k = 32k = 64k = 128k = 256
A2 satisfied60.0%75.0%77.5%83.1%83.0%
Agreement when teacher correct100.0%95.0%93.6%93.3%94.4%

Table 2: Empirical evaluation of Assumption A2: Teacher-student agreement on factual and counterfactual pairs. Results are reported on TriviaQA, averaged across five random seeds and different few-shot training budgets. A2 is satisfied when the student exactly matches the teacher on both factual and counterfactual samples. Conditional agreement measures teacher-student agreement when the teacher correctly answers both samples.

A3 (Boundary Coverage): We provide geometric motivation for Assumption A3 through two properties that make meaningful boundary coverage plausible under limited supervision:

  • Lower-dimensional manifold of language representations. Prior studies suggest that natural-language representations may lie on lower-dimensional manifolds despite being embedded in high-dimensional spaces [1, 2]. Thus, boundary alignment is primarily relevant where meaningful inputs occur, rather than throughout the entire representation space, reducing the effective complexity of the boundaries that need to be covered.

  • Smooth decision boundaries and knowledge distillation. Prior work suggests that knowledge distillation transfers decision-boundary information [3, 4] and exhibits statistical and regularization properties [5–7]. These findings motivate the expectation that teacher and student boundaries may remain smoothly aligned even between observed training regions. Specifically, smooth boundaries may remain close in uncovered regions, whereas a highly oscillating student boundary could deviate substantially without direct supervision. Knowledge distillation may encourage smooth alignment between teacher and student boundaries rather than large deviations in uncovered regions, making meaningful boundary coverage more plausible with limited samples. We illustrate both scenarios in Figure 6 of Appendix A.1. Additionally, our empirical results show that teacher-student agreement generally increases with the training budget, consistent with improved boundary alignment.

Together, these arguments provide geometric motivation for A3 without requiring dense coverage of the entire representation space.

Overall, our empirical analyses support the small-perturbation assumption (A1) and demonstrate substantial teacher-student agreement on factual-counterfactual pairs (A2). For A3, the manifold structure of language representations and the boundary-transfer properties of knowledge distillation provide theoretical motivation. Together, these findings motivate the assumptions underlying Proposition 1 and clarify the geometric role of counterfactual supervision. We have added this analysis in Appendix A.1 and revised Section 2.2 of the manuscript to address these concerns.

[1] Lee et al. (2025). Shared global and local geometry of language model embeddings. COLM.

[2] Tulchinskii et al. (2023). Intrinsic dimension estimation for robust detection of AI-generated texts. NeurIPS.

[3] Heo et al. (2019). Knowledge transfer via distillation of activation boundaries formed by hidden neurons. AAAI.

[4] Ojha et al. (2023). What knowledge gets distilled in knowledge distillation? NeurIPS.

[5] Menon et al. (2021). A statistical perspective on distillation. ICML.

[6] Zhou et al. (2021). Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. ICLR.

[7] Mobahi et al. (2020). Self-distillation amplifies regularization in Hilbert space. NeurIPS.

(continued in the next comment.)

(continuing ..)

Part 2 (W2 & Q2): Statistical Significance of Performance Improvements

COFFEQA achieves statistically significant and overall improvements over KD. We conducted additional statistical tests using paired comparisons across five seeds, reporting 95% confidence intervals and paired t-tests, as shown in Table 3. For a fair comparison, both methods use the same selected factual examples and random seeds. Averaged across five datasets, COFFEQA achieves statistically significant improvements of 4.1 EM points [3.4, 4.8] (p < 0.001) and 2.8 F1 points [2.5, 3.2] (p < 0.001) over KD. COFFEQA outperforms KD across all five seeds, with statistically significant overall improvements across training budgets. Furthermore, COFFEQA achieves higher EM in 24 of the 25 dataset–budget combinations, demonstrating consistent gains across datasets and training budgets. We have included this detailed analysis in Appendix A.2 of the revised manuscript.

kEM gain [95% CI]p-value (EM)F1 gain [95% CI]p-value (F1)Seeds with Higher EM
16+1.5 [−0.4, 3.3]0.091+1.5 [0.2, 2.7]0.0304/5
32+3.4 [0.7, 6.1]0.026+2.4 [1.1, 3.7]0.0075/5
64+8.1 [5.1, 11.1]0.002+4.8 [2.9, 6.8]0.0025/5
128+3.6 [0.0, 7.2]0.049+2.6 [0.4, 4.8]0.0295/5
256+3.9 [1.0, 6.9]0.021+3.0 [1.5, 4.5]0.0055/5
All k+4.1 [3.4, 4.8]<0.001+2.8 [2.5, 3.2]<0.0015/5

Table 3: Statistical significance of COFFEQA improvements over KD on T5. EM and F1 gains are averaged across five datasets and evaluated using paired t-tests over five random seeds, with 95% confidence intervals. Positive gains indicate improvements over KD. Blue shading highlights aggregate results.

Part 3 (W2 & Q2): Computational Cost Analysis of Counterfactual Generation and Validation

Counterfactual generation introduces a manageable, one-time computational cost, with minimal overhead from CFD validation. We conducted a detailed cost analysis covering all generated counterfactual candidates, including those filtered during validation. Table 4 reports the number of candidates examined, generation time, validation time, and API costs required to obtain 128 accepted counterfactuals per dataset. We measured each step separately to provide a comprehensive estimate of the overall computational cost.

DatasetFactuals examinedRejected after 5 triesTotal candidatesGPT Generation Time (s)CFD Validation Time (s)API cost (USD)
NQ267130106595002510.28
SQuAD163264391832940.12
NewsQA2007179481861680.21
TriviaQA4712161394344102780.37
HotpotQA1703753638431140.14

Table 4: Counterfactual generation and validation costs across five datasets. Results are reported for generating 128 accepted counterfactuals per dataset including rejected candidates.

Most of the additional computational cost comes from GPT-based counterfactual generation, while CFD validation contributes only a small fraction of the total processing time. The student distillation step is slightly faster than standard KD as shown in Appendix G.3 of the original manuscript. Importantly, counterfactual generation is performed only once per dataset, and the accepted counterfactuals are reused across multiple few-shot budgets and random seeds, allowing the generation cost to be amortized across experiments. We have included this detailed analysis in Appendix A.5 of the revised manuscript.

(continued in the next comment.)

Comments on the paper

1 comment

We sincerely thank the Area Chair and all reviewers for their careful evaluation, constructive feedback, and valuable suggestions. We appreciate that Reviewer 1 recognized COFFEQA as "a meaningful contribution to few-shot QA distillation" and highlighted its "broad empirical evaluation." We are also encouraged that Reviewer 2 described CFD as "an intuitive and potentially reusable heuristic for synthetic data filtering" and acknowledged the practical value of our approach. We further thank Reviewer 3 for recognizing that our work addresses "an important problem" and that using counterfactual examples to expose changes in teacher behavior is "intuitively meaningful."

In response to the reviewers' comments, we have substantially strengthened the manuscript through additional experiments, theoretical clarification, and expanded analyses. The key revisions are summarized below.

  1. Stronger baselines and independent contributions (Appendix A.3). We evaluated COFFEQA against random, medium-margin, and large-margin factual sampling baselines, as well as simple entity replacement. These experiments demonstrate the additional benefits of counterfactual supervision beyond factual sample selection and highlight the advantages of LLM-based counterfactual generation.

  2. Statistical significance and robustness (Appendix A.2). We conducted paired statistical tests across five random seeds, reporting 95% confidence intervals and significance levels. COFFEQA achieves statistically significant overall improvements over standard KD.

  3. Theoretical clarification and empirical analysis (Appendix A.1 & Section 2.2). We refined the geometric formulation of context-response boundaries for generative QA and clarified the assumptions underlying Proposition 1. We also added empirical analyses of counterfactual perturbation sizes and teacher–student agreement, together with geometric motivation for boundary coverage.

  4. Counterfactual validation and quality (Appendices A.6.1 and A.6.2). We conducted component-wise ablations of CFD, comparing Answer Agreement (AA), Contextual Loss Ratio (CLR), and their combination. We further evaluated counterfactual validity, contextual consistency, and minimality through quantitative measures and human and LLM-based assessments.

  5. Generalization to newer model families and teacher sizes (Appendix A.4). We extended our evaluation to Qwen3-8B/Qwen3-1.7B and a substantially larger Qwen2.5-72B teacher. COFFEQA achieves higher average EM than KD overall, supporting its applicability across model families and teacher scales.

  6. Additional distillation objectives (Appendix A.6.3). We investigated incorporating a DPO-style contrastive loss into COFFEQA and compared its performance with standard KD and the original COFFEQA formulation.

  7. Computational cost and generator choice (Appendices A.5 and A.6.2). We expanded the cost analysis to account for counterfactual generation attempts, rejected candidates, CFD validation, and API expenses. We also compared GPT-4o-mini and GPT-4o, demonstrating the trade-off between downstream performance and generation cost.

Together, these additions provide a more comprehensive assessment of COFFEQA's effectiveness, independent contributions, generalizability, and computational efficiency. We have incorporated the corresponding results and clarifications into the revised manuscript and provide detailed responses to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.