SynthFailMD: Failure by Design for Synthetic Multi-Document Generation

Enterprise knowledge agents increasingly operate over collections of documents with diverse modalities and artifacts, such as text, tables, charts, forms, images, and annotations. When these agents fail, the failures are often systematic, recurring under similar document and artifact conditions and providing a useful signal for evaluation. However, the enterprise documents underlying these failures are often private and cannot be reproduced as open benchmarks. We introduce SynthFailMD, a synthetic multi-document benchmark generator that constructs targeted document failure conditions without exposing private enterprise documents. SynthFailMD generates two connected documents at a time, along three dimensions: (1) document realism, using information graphs extracted from UniDoc-Bench across five domains to ground cross-document topics and relationships; (2) visual realism, using document recipes derived from real document distributions of tables, charts, text, and other artifacts in enterprise PDFs; and (3) behavioral realism, using 27 target failure types spanning seven artifact categories, including text, tables, charts, forms, images, annotations, and OCR. A document critique-and-repair (DCR) agent further reduces answer leakage, contradictions, and inconsistencies in generated documents. Using SynthFailMD, we construct 410 PDFs and approximately 3,798 question-answer pairs. Across matched control and failure-conditioned documents, accuracy drops from 70.70% to 57.06%, showing that targeted failure conditions produce measurable degradation. SynthFailMD provides a scalable, reproducible, and privacy-preserving approach for evaluating enterprise document intelligence systems.