- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces FondueQA, a benchmark for studying how language models use internal knowledge to answer questions when retrieval evidence is incomplete.
Critical strengths
- The dataset provides a substantial collection of questions across several evidence structures, supporting comparisons under different evidence conditions.
- Studying how models compensate for missing evidence offers an interesting perspective on the interaction between internal knowledge and retrieval.
Critical weaknesses
- The lack of independent human validation leaves answer correctness, evidence support, and the claimed isolation of required facts insufficiently verified.
- Despite evaluating several open model families, the absence of commercial flagship models leaves unclear whether stronger systems would find this benchmark too easy.
- Results are mostly presented in figures, making exact comparisons difficult. Please tabulate absolute accuracy across models and evidence conditions, clearly specifying denominators.
- The text within the figures is too small, reducing readability and visual clarity. Please enlarge it throughout the paper.
Questions for authors
- Could two annotators independently audit a representative subset, such as 20%, and report agreement, error rates, and disagreement resolution?
- How do representative commercial flagship models perform under the same evidence conditions, and do the main findings hold for them?
- Would a small reinforcement learning study improve a local model toward 80% accuracy on separate test questions under clearly specified evidence conditions?
3 comments
Thank you for the response. My concerns remain unresolved, so I will maintain my original score. In addition, little suggestion, even if you cannot conduct the experiments I requested, I would still encourage you to address my questions through alternative experiments. Otherwise, your response is unlikely to convince me and other reviewers.
Thank you for your follow-up and for encouraging us to explore additional experiments. We have since conducted two further evaluations and would like to share the results.
Additional model evaluation. We agree and appreciate the option for reporting alternative experiments under similar conditions, and evaluate GLM 5.3 as a strong open-weight alternative to commercial flagship models. According to the LLM Stats reasoning leaderboard, it currently ranks 20th overall and third among open-weight models. In the following, we share results obtained within our available compute budget, using gpustack/GLM-5.3-W4A8:
| Question type | n | All-Gold | Partial A | Partial B | No-Gold |
|---|---|---|---|---|---|
| Two-hop Bridge | 151 | 79.5% | 60.0% | 54.1% | 27.2% |
| Comparison | 223 | 86.1% | 64.4% | 57.1% | 53.4% |
| Contrastive | 40 | 81.7% | 58.3% | 65.0% | 62.5% |
| Cross-Property | 39 | 100.0% | 94.0% | 80.3% | 92.3% |
These preliminary results cover 453 questions for which predictions have been computed across all four evidence conditions. The results show that performance varies considerably with the available evidence. Particularly, for two-hop bridge questions, accuracy drops from 79.5% in the All-Gold setting to 27.2% in No-Gold. These findings suggest that even a strong open-weight model struggles when supporting evidence is missing.
RL experiment. Following your suggestion, we also implemented a small-scale GRPO-LoRA study with Gemma-4-E2B and Gemma-4-E4B, training on full- and partial-evidence conditions. Given the limited resources during the rebuttal, we evaluated the models after 68 and 51 training steps, respectively.
We evaluated both models on 986 held-out questions, ensuring that no required facts overlap between the training and held-out sets. Mean partial-evidence EM increased from 34.79% to 36.41% for E2B (+1.62 pp) and from 40.82% to 42.55% for E4B (+1.73 pp).
We take this small-scale experiment indeed as an indicator that training under these evidence conditions can improve an LLM's behavior. Although the focus of our paper is on the evaluation and providing an analysis on the parametric compensation capabilities, we assess the model optimization problem motivated by the RL experiment as a fruitful future direction.
We appreciate that the reviewer agrees that our “substantial collection across evidence structures” supports the investigation of parametric compensation. Moreover, the reviewer’s feedback helped us to improve the paper’s presentation, strengthen its validity through human annotation, additional experiments with a frontier open-source model and an initial RL study highlighting a promising direction for future work, potentially leveraging our proposed benchmark. We hope that we have satisfactorily addressed all of the reviewer's concerns and that the additional results and clarifications provide a stronger basis for reassessing the paper's contributions. We would welcome any further feedback or questions and remain open to additional discussion.
Thank you very much for your thoughtful feedback and for recognizing the careful construction of FondueQA and its potential for studying model behavior under incomplete evidence.
W1 & Q1: Human annotation
We agree that independent human validation is important. We therefore audited 500 randomly sampled questions from the final release. Each question was independently reviewed by two annotators from a pool of six, covering the question, answer, required facts, and complete context.
Overall, 94.3% of the 1,000 individual judgments were Valid, and both annotators judged 89.6% of questions valid. Most flagged concerns involved other issues, like ambiguous questions or incomplete answers. Direct leakage was flagged by at least one annotator in only 8 questions (1.6%) and inferential leakage in 9 (1.8%).
Valid/Error agreement was 90.6%, with Krippendorff’s α = 0.126 (95% bootstrap CI: −0.006 to 0.272). Most judgments were Valid, which contributes to the low chance-corrected agreement despite 90.6% raw agreement. The large majority of questions received valid judgments, and leakage flags were uncommon. These findings support the effectiveness of our fact-isolation procedure. We will include the audit results and agreement statistics in the revised manuscript.
W2 & Q2: Flagship LLMs
We agree that evaluating commercial flagship models would provide a valuable complementary perspective on the generality of our findings. However, the main FondueQA context-conditioned evaluation alone requires approximately 130,000 requests per model, before additional atomic-probe and candidate-competition evaluations. Running the full evaluation on commercial APIs would therefore incur substantial costs that exceed our available monetary budget.
W3 & W4: Tabularization & Presentation
Thank you for pointing out the presentation issues. We will add a table, reporting absolute Exact Match scores and explicit evaluation denominators. This complements the relative comparisons with absolute performance across the key evidence conditions, including No-Gold, All-Gold, Bridge-only, and Target-only. We are also revising the figures throughout the manuscript to improve readability, particularly by increasing font sizes, axis labels, and legends.
Q3: RL baseline
Thank you for the suggestion. For the scope of this work, we focus on evaluating and understanding parametric compensation rather than improving model performance through additional training. We therefore consider the proposed RL experiment beyond the scope of the current submission.
We sincerely thank the reviewer for their constructive feedback and for the opportunity to further strengthen the empirical evaluation and argumentation of our paper. We hope that our responses adequately address the concerns raised and would welcome further discussion of any remaining questions or concerns.