- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces a synthetic benchmark for studying hallucination and abstention under missing document evidence. It builds structured latent worlds, renders facts into documents, masks evidence, and recomputes answerability using fact provenance and inference dependencies. The benchmark covers four enterprise scenarios and evaluates nine LLMs with RAG and CLI agents. The main finding is that strong performance on answerable questions does not imply reliable abstention when evidence is missing.
Critical strengths
- The provenance-aware framework can mask documents and automatically recompute which questions remain answerable, enabling systematic study of abstention under incomplete evidence.
- The paper evaluates multiple recent models under both RAG and CLI settings and across QA and structured prediction tasks, providing a useful empirical comparison of answering and abstention behavior.
Critical weaknesses
- The framework novelty appears limited to me. The ideas of synthetic lated world, multihop inference graphs, evidence masking are already well established. And importantly, the paper does not empirically compare against other factuality/hallucination benchmarks (incomplete-information QA, controlled-evidence-removal). It is unclear whether this benchmark reveals qualitatively new failure modes or model rankings beyond what existing benchmarks already capture.
- The paper’s central claim is that provenance and inference dependencies enable exact recomputation of answerability after document masking. However, this exactness is defined relative to the authored symbolic specification and its inference rules. The generated natural-language corpus may contain implicit, redundant, or commonsense evidence that is not represented in the simulator’s dependency graph. Consequently, an item labeled unanswerable by the framework may still be answerable from the visible documents through a valid inference path not encoded in the specification. This concern is acknowledged indirectly by the authors, who note that they do not establish whether humans could answer every item labeled unanswerable from the remaining evidence. This substantially weakens the claim of “exact” evidence control.
- The paper validates structural consistency of the specification and simulator, and asks domain experts to review each specification and one generated lane. However, it does not systematically measure whether the generated latent worlds are semantically realistic, whether the fact ontology is sufficiently complete, or whether the authored inference rules capture the valid inferences available in the domain. Since downstream answerability labels depend directly on this symbolic world, the lack of a stronger quality evaluation weakens the benchmark’s validity.
- nit: figure 1 is difficult to understand
Questions for authors
- How do the authors validate the semantic completeness of the latent world and inference rules, beyond structural checks and limited expert review? How do we measure the quality of the pipeline.
- Have the authors tested whether humans can answer questions labeled unanswerable from the remaining visible documents?
- How does this benchmark compare empirically with the closest abstention or controlled-evidence-removal benchmarks, and what unique failure modes does it reveal? Also methodology like MiniCheck (Tang et al., 2024)