- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses an imminent problem in analyzing failures of confidential enterprise knowledge agents through document synthesis. To tackle this problem, the authors propose SynthFailMD, a synthetic multi-document benchmark generator that constructs targeted document failure conditions without exposing private enterprise documents. Specifically, SynthFailMD synthesizes two related documents at a time along three dimensions: document realism, visual realism, and behavioral realism. As a result, SynthFailMD generates 410 PDFs with around 3,798 question-answer pairs, and experiments show that targeted failure conditions produce measurable degradation.
Critical strengths
This paper aims to address an important and interesting problem in analyzing failures of enterprise knowledge agents through document synthesis, as such failures are often hard to reproduce or inaccessible.
Critical weaknesses
- The experiments are only conducted on the GPT model family. Including LLMs from other model families, such as Claude, Gemini, and Qwen, might further strengthen the claims in the paper.
- As discussed in the conclusion, some document failures arise only when evidence is rooted in three or more documents. What extension of SynthFailMD is needed to support this scenario? Moreover, what aspects of the current findings can reasonably generalize to these cases?
- The paper reports automated consistency checks and LLM-based verification, while no human assessment of the generated documents and QA pairs. Agreement among LLM judges does not establish dataset correctness. An additional human annotation of answerability, reference-answer correctness, and unintended generation errors would strengthen the benchmark’s validity. Minors:
- I find SynthFailMd interesting and would like to see some generated documents. However, Figure 5 in the appendix is too small to read. I recommend that the authors include generated documents in the supplemental materials.
- Section 4.2 describes Canonical Markdown as content-preserving, while Section 5.1 notes that some visual information is not preserved. What evidence is actually provided for rasterized text, charts, handwriting, and annotations?
Questions for authors
See above.
2 comments
Summary
We thank the reviewer for the detailed feedback and constructive suggestions. We hope our responses address the concerns regarding (1) generalizability across model families, (2) extending the framework to three or more documents, (3) human validation, (4) supplementary materials, and (5) the distinction between Canonical Markdown and Direct-PDF VLM evaluation.
We greatly appreciate the reviewer's suggestions. Thank you again!
Thank you very much for your thoughtful suggestions. We address the comments below:
[Comment 1] Generalizability across model families
We thank the reviewer for this suggestion. We have expanded our evaluation beyond the initial 3 GPT models to include GPT-5.6 Luna, GPT-5.6 Terra, Kimi K2.5, Llama 4 Maverick, and Qwen 3.5 27B. As shown in revised Table 3, all five additional models exhibit performance degradation across both Canonical Markdown and VLM settings. Importantly, this includes three non-GPT models (Kimi K2.5, Llama 4 Maverick, and Qwen 3.5 27B), providing evidence that the observed degradation is not specific to the GPT model family.
[Comment 2] Extending to three or more documents
We thank the reviewer for this question. Extending SynthFailMD to three or more documents would primarily require modifying the inter-document information flow architecture. Specifically, for three documents, we would construct three relational tables connected through a common key, enabling information dependencies across all three documents. For additional documents, the number of interconnected relational tables would increase accordingly. Topic seeds can still be derived from the information graph, while the remaining generation and failure-injection pipeline would require relatively minor modifications. We expect the general finding that injected failures degrade QA performance to extend to these settings, although the magnitude of degradation may vary with more complex cross-document dependencies. We consider this an important future direction that we intend to actively pursue.
[Comment 3] Human validation
We thank the reviewer for highlighting this important limitation. We acknowledge that automated consistency checks and LLM-based verification alone do not fully establish dataset correctness. While we have not yet included a human evaluation in the revised manuscript, we are actively designing a human validation study to assess QA answerability, reference-answer correctness, unintended generation errors, and the realism of synthetic failure conditions relative to real-world enterprise document-understanding failures. We plan to incorporate this study and its findings into the final revised manuscript.
[Comment 4] Supplementary materials
We have added supplementary PDF materials for the finance domain, covering all 27 RCA types. 14 RCA types are paired and include both control and treatment documents (two each). The remaining 13 are treatment-only, with two treatment documents each, since they have no paired control arm. In total, the supplement contains 82 PDFs. We have also uploaded the revised rebuttal PDF, with all changes highlighted for ease of reference. Thank you!
[Comment 5] Evidence for rasterized text, charts, handwriting and annotations
We thank the reviewer for identifying this ambiguity. We clarify that content-preserving in Section 4.2 refers to preserving the textual and structured information represented in Canonical Markdown, rather than guaranteeing preservation of all visual information in the original PDFs. Rasterized text, charts, handwriting, and annotations may contain information that cannot be fully represented in Markdown. Our VLM evaluation retains the rendered PDF pages and therefore provides a complementary setting for assessing failures involving visual information. We have revised Sections 4.2 and 5.1 to make this distinction explicit. As shown in Appendix A.19 Table 25 (selected results below), the two settings exhibit different performance patterns across visually grounded failure types.
Selected results from Appendix A.19 Table 25 (accuracy, %).