acceptodds
Under review as a conference paper at ICLR 2027

SynthFailMD: Failure by Design for Synthetic Multi-Document Generation

Abstract

Enterprise knowledge agents increasingly operate over collections of documents with diverse modalities and artifacts, such as text, tables, charts, forms, images, and annotations. When these agents fail, the failures are often systematic, recurring under similar document and artifact conditions and providing a useful signal for evaluation. However, the enterprise documents underlying these failures are often private and cannot be reproduced as open benchmarks. We introduce SynthFailMD, a synthetic multi-document benchmark generator that constructs targeted document failure conditions without exposing private enterprise documents. SynthFailMD generates two connected documents at a time, along three dimensions: (1) document realism, using information graphs extracted from UniDoc-Bench across five domains to ground cross-document topics and relationships; (2) visual realism, using document recipes derived from real document distributions of tables, charts, text, and other artifacts in enterprise PDFs; and (3) behavioral realism, using 27 target failure types spanning seven artifact categories, including text, tables, charts, forms, images, annotations, and OCR. A document critique-and-repair (DCR) agent further reduces answer leakage, contradictions, and inconsistencies in generated documents. Using SynthFailMD, we construct 410 PDFs and approximately 3,798 question-answer pairs. Across matched control and failure-conditioned documents, accuracy drops from 70.70% to 57.06%, showing that targeted failure conditions produce measurable degradation. SynthFailMD provides a scalable, reproducible, and privacy-preserving approach for evaluating enterprise document intelligence systems.

Primary area:
datasets and benchmarks
Keywords:
Multimodal document understanding, document intelligence, synthetic data generation, benchmark generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S4rPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses an imminent problem in analyzing failures of confidential enterprise knowledge agents through document synthesis. To tackle this problem, the authors propose SynthFailMD, a synthetic multi-document benchmark generator that constructs targeted document failure conditions without exposing private enterprise documents. Specifically, SynthFailMD synthesizes two related documents at a time along three dimensions: document realism, visual realism, and behavioral realism. As a result, SynthFailMD generates 410 PDFs with around 3,798 question-answer pairs, and experiments show that targeted failure conditions produce measurable degradation.

Critical strengths

This paper aims to address an important and interesting problem in analyzing failures of enterprise knowledge agents through document synthesis, as such failures are often hard to reproduce or inaccessible.

Critical weaknesses
  1. The experiments are only conducted on the GPT model family. Including LLMs from other model families, such as Claude, Gemini, and Qwen, might further strengthen the claims in the paper.
  2. As discussed in the conclusion, some document failures arise only when evidence is rooted in three or more documents. What extension of SynthFailMD is needed to support this scenario? Moreover, what aspects of the current findings can reasonably generalize to these cases?
  3. The paper reports automated consistency checks and LLM-based verification, while no human assessment of the generated documents and QA pairs. Agreement among LLM judges does not establish dataset correctness. An additional human annotation of answerability, reference-answer correctness, and unintended generation errors would strengthen the benchmark’s validity. Minors:
  4. I find SynthFailMd interesting and would like to see some generated documents. However, Figure 5 in the appendix is too small to read. I recommend that the authors include generated documents in the supplemental materials.
  5. Section 4.2 describes Canonical Markdown as content-preserving, while Section 5.1 notes that some visual information is not preserved. What evidence is actually provided for rasterized text, charts, handwriting, and annotations?
Questions for authors

See above.

2 comments

Thank you very much for your thoughtful suggestions. We address the comments below:

[Comment 1] Generalizability across model families

We thank the reviewer for this suggestion. We have expanded our evaluation beyond the initial 3 GPT models to include GPT-5.6 Luna, GPT-5.6 Terra, Kimi K2.5, Llama 4 Maverick, and Qwen 3.5 27B. As shown in revised Table 3, all five additional models exhibit performance degradation across both Canonical Markdown and VLM settings. Importantly, this includes three non-GPT models (Kimi K2.5, Llama 4 Maverick, and Qwen 3.5 27B), providing evidence that the observed degradation is not specific to the GPT model family.

ModelMarkdown ControlMarkdown TreatmentVLM ControlVLM Treatment
GPT-5.6 Sol73.21%57.62%79.49%72.58%
GPT-5.6 Luna69.98%53.99%73.44%68.04%
GPT-5.6 Terra72.05%56.38%74.37%67.91%
GPT-5.571.32%55.22%78.94%73.74%
GPT-5.470.06%55.12%78.56%71.57%
Kimi K2.565.10%52.35%67.63%64.68%
Llama 4 Maverick61.25%47.64%28.96%25.19%
Qwen 3.5 27B67.88%50.88%74.64%72.17%
Overall68.86%53.65%69.50%64.49%

[Comment 2] Extending to three or more documents

We thank the reviewer for this question. Extending SynthFailMD to three or more documents would primarily require modifying the inter-document information flow architecture. Specifically, for three documents, we would construct three relational tables connected through a common key, enabling information dependencies across all three documents. For additional documents, the number of interconnected relational tables would increase accordingly. Topic seeds can still be derived from the information graph, while the remaining generation and failure-injection pipeline would require relatively minor modifications. We expect the general finding that injected failures degrade QA performance to extend to these settings, although the magnitude of degradation may vary with more complex cross-document dependencies. We consider this an important future direction that we intend to actively pursue.


[Comment 3] Human validation

We thank the reviewer for highlighting this important limitation. We acknowledge that automated consistency checks and LLM-based verification alone do not fully establish dataset correctness. While we have not yet included a human evaluation in the revised manuscript, we are actively designing a human validation study to assess QA answerability, reference-answer correctness, unintended generation errors, and the realism of synthetic failure conditions relative to real-world enterprise document-understanding failures. We plan to incorporate this study and its findings into the final revised manuscript.


[Comment 4] Supplementary materials

We have added supplementary PDF materials for the finance domain, covering all 27 RCA types. 14 RCA types are paired and include both control and treatment documents (two each). The remaining 13 are treatment-only, with two treatment documents each, since they have no paired control arm. In total, the supplement contains 82 PDFs. We have also uploaded the revised rebuttal PDF, with all changes highlighted for ease of reference. Thank you!


[Comment 5] Evidence for rasterized text, charts, handwriting and annotations

We thank the reviewer for identifying this ambiguity. We clarify that content-preserving in Section 4.2 refers to preserving the textual and structured information represented in Canonical Markdown, rather than guaranteeing preservation of all visual information in the original PDFs. Rasterized text, charts, handwriting, and annotations may contain information that cannot be fully represented in Markdown. Our VLM evaluation retains the rendered PDF pages and therefore provides a complementary setting for assessing failures involving visual information. We have revised Sections 4.2 and 5.1 to make this distinction explicit. As shown in Appendix A.19 Table 25 (selected results below), the two settings exhibit different performance patterns across visually grounded failure types.

Selected results from Appendix A.19 Table 25 (accuracy, %).

Failure typeMarkdown ControlMarkdown TreatmentVLM ControlVLM Treatment
Scanned chart recognition30.9528.5758.1652.72
Scanned text recognition68.2334.8973.8855.95
Handwritten content55.4339.8160.9559.81
Annotation understanding75.3738.4372.4053.29

Summary

We thank the reviewer for the detailed feedback and constructive suggestions. We hope our responses address the concerns regarding (1) generalizability across model families, (2) extending the framework to three or more documents, (3) human validation, (4) supplementary materials, and (5) the distinction between Canonical Markdown and Direct-PDF VLM evaluation.

We greatly appreciate the reviewer's suggestions. Thank you again!

Reviewer_CK4SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework for generating synthetic multi-document benchmarks with targeted failure conditions. The framework combines information graphs extracted from real documents, document recipes based on observed layout and artifact distributions, and failure-conditioned generation. It also uses a Document Critique-and-Repair agent to reduce unintended inconsistencies and answer leakage. Experiments on several language models and document parsers show that the injected conditions reduce QA accuracy. The authors further evaluate document diversity, visual similarity to real documents, and the effects of DCR.

Critical strengths
  1. The paper addresses a relevant problem: enterprise documents are often private, making it difficult to reproduce and study document-agent failures. The matched control-treatment design allows to examine how specific document conditions affect model performance while keeping the underlying information fixed.
  2. The combination of information-graph grounding, document recipes, and failure-conditioned generation is useful for constructing connected documents with different artifacts. The authors also make an effort to control generation errors through DCR. The experiments cover multiple models and document parsers, and the overall accuracy degradation provides initial evidence that the generated failure conditions are effective.
Critical weaknesses
  1. Although the authors derive document recipes from real data and show visual overlap in Figure 3, Appendix A.17 reports a C2ST AUC of 0.997 when distinguishing real and synthetic documents. This suggests a substantial distribution gap. More importantly, the paper does not convincingly establish whether the generated failure conditions reflect the types of failures encountered in actual enterprise workflows. Without stronger validation, it is difficult to assess how well the benchmark results generalize to real documents.
  2. Although the paper discusses recent approaches such as MDBench, SynthDocBench, and DocDjinn, the comparisons are mainly qualitative. Direct experimental comparisons are missing, making it difficult to assess the advantages of SynthFailMD over existing methods. In addition, cross-family evaluations only use treatment documents, limiting the evidence for generalization.
Questions for authors
  1. How should the near-perfect real-versus-synthetic classification result (C2ST AUC = 0.997 in Table 19) be interpreted alongside the paper's realism claims? Is there any human evaluation or comparison with real enterprise failure cases that supports the practical relevance of the generated documents?
  2. Given that each paired failure type contains only five document ecosystems, how stable are the observed effects across different instances? Would the authors expect similar results with a larger number of independently generated document pairs? Also, Table 8 lists highlighted text as a paired condition, whereas Figures 7 and 8 report annotations instead. Could the authors clarify this discrepancy?
  3. Appendix A.15 reports 76% agreement among the three LLM judges on the evaluation subset. Have the authors checked whether the main control-treatment conclusions remain consistent when using alternative judges? This would help establish whether the measured degradation is robust to the choice of evaluator.
4 comments

Thank you for your thoughtful suggestions. We address the comments below:

[Rebuttal Part 1/2]

[Comment 1, Question 1] Realism and validity of synthetic failure

Document Realism and Distributional Gap: We acknowledge that the C2ST AUC of 0.997 (Appendix A.18, Table 24) indicates a distributional gap between real and synthetic documents. However, our complementary enterprise-domain classification experiment (Appendix A.18, Table 23) achieves ROC-AUC 0.9622, with 96.8% of SynthFailMD documents classified as enterprise-like, suggesting that the generated documents capture characteristics associated with the selected enterprise reference distribution, despite remaining distinguishable from real documents.

Validity of Failure Mechanisms: The development of SynthFailMD was grounded in real enterprise document-understanding failures identified and annotated through root cause analysis (RCA) in enterprise systems (Appendix A.5, Table 9). These observed failure categories informed the construction of our failure taxonomy and the design of corresponding synthetic failure conditions. Thus, the targeted failure mechanisms reflect challenges commonly encountered in real-world enterprise document-understanding workflows. To further validate the induced failures, we conducted an additional failure attribution analysis (Section 7, Table 4), providing evidence that the generated conditions induce effects consistent with their intended failure mechanisms.

Evaluation SettingAttributed / C→WAttribution Rate
Canonical Markdown610/72284.5%
Direct-PDF VLM236/34967.6%
Overall846/1,07179.0%

Human Validation: While we have not yet conducted a formal human evaluation, we are actively designing a validation study to assess the realism and practical relevance of synthetic failure conditions against real-world enterprise document-understanding failures. We plan to incorporate its findings into the final revised manuscript.


[Comment 2] Comparison with existing baseline and cross-family evaluation

Existing Baseline: While MDBench, SynthDocBench, and DocDjinn all generate synthetic documents, their primary objectives and evaluation methodologies differ from SynthFailMD. To clarify these distinctions, we provide an expanded comparison below.

DimensionMDBenchSynthDocBenchDocDjinnSynthFailMD (Ours)
Primary objectiveMulti-document reasoning evaluationLong-context visual document understandingSynthetic training-data generation for document AIFailure-oriented multi-document evaluation
Document settingText-based multi-document setsLong, visually rich PDF reportsVisually realistic, primarily individual documentsHeterogeneous, connected multi-document ecosystems
Controlled variablesKnowledge structure and reasoning challengesLength, layout, modality, question typeSeed-conditioned content, layout, and visual appearanceRCA-derived failure conditions
Purpose of model evaluationMeasure multi-document reasoning capabilitiesDiagnose sensitivity to document and question characteristicsMeasure synthetic-data training utilityMeasure sensitivity to document-induced failures
Paired control/treatmentNot a primary evaluation designNot a primary evaluation designNot a primary evaluation designSupported for 14 failure types
Evaluation focusMulti-document reasoning analysisSensitivity to document characteristicsVisual fidelity and downstream training utilityFailure-conditioned performance degradation and attribution analysis

Unlike existing approaches, SynthFailMD targets RCA-derived failure mechanisms in interconnected, heterogeneous document ecosystems through paired control–treatment evaluation. Given differences in generation objectives and evaluation protocols, directly comparing QA accuracies would not be an eye-to-eye comparison. We acknowledge this limitation and provide a methodological comparison to clarify SynthFailMD's distinct contributions.

Rebuttal Part [2/2]

Cross-Family Evaluation: We thank the reviewer for this observation. We have added the evaluation to include paired control–treatment documents across six models. Table 3 shows statistically significant degradation across all models for the 14 paired failure types (12.76–17.00 pp for canonical Markdown; 2.47–6.99 pp for VLM). Additionally, Appendix A.15 (Tables 18–19) reports cross-family evaluations across all 27 failure types and 410 PDFs, showing positive correlations in failure-type difficulty rankings (Spearman's ρ = 0.608–0.878). These results provide further evidence of cross-family consistency in failure-induced degradation and difficulty patterns.


[Question 2] Ecosystem level stability

We thank the reviewer for raising this concern. To assess the stability of the observed effects across document ecosystems, we conducted an additional leave-one-ecosystem-out (LOEO) sensitivity analysis across all 14 paired failure types and five domains. We used the direct-PDF VLM evaluation with three models (GPT-5.6 Sol, GPT-5.5, and GPT-5.4), considering only matched control–treatment question pairs.

Table: Ecosystem-level stability of failure-induced degradation under leave-one-ecosystem-out analysis (direct-PDF VLM). Positive Δ indicates lower accuracy on treatment documents. All values are in percentage points (pp).

Failure TypeMean ΔEcosystem Δ RangeLOEO Range
Scanned Chart Recognition7.182.17 to 23.813.02 to 8.43
Scanned Text Recognition11.421.85 to 14.4110.67 to 13.81
Scanned Table Recognition5.890.46 to 20.372.27 to 7.25
Handwritten Content5.34−5.21 to 13.893.21 to 7.98
Small/Graphical Marks3.73−0.62 to 7.972.67 to 4.82
Annotations19.3611.11 to 25.0017.95 to 21.42
Relational Reasoning−0.37−2.90 to 1.39−0.81 to 0.26
Merged Cell Handling4.281.85 to 6.373.76 to 4.89
Cross-Page Tables−0.99−3.47 to 1.23−1.55 to −0.37
Column Alignment0.610.00 to 1.560.37 to 0.76
Blank/Missing Values16.895.36 to 30.5613.48 to 19.78
Cross-Page Reasoning5.901.75 to 10.324.79 to 6.93
Long-Context Retrieval2.700.00 to 7.141.59 to 3.37
Reading Order Preservation8.14−0.88 to 18.425.56 to 10.39

The results show that 12 of 14 failure types exhibit positive mean degradation, and all 12 retain positive mean degradation under every single-ecosystem exclusion. This suggests that the observed positive average effects are not driven by any single document ecosystem. While the degradation magnitude varies across ecosystems, the LOEO analysis provides evidence of robustness within the evaluated ecosystems. Further methodological details and results are provided in Appendix A.20.


[Question 2] Table 8 discrepancy

We thank the reviewer for identifying this inconsistency. This was an error in Table 8 (new Table 9): annotations rather than highlighted text, is the intended paired failure condition. We have corrected the table, rerun the evaluations using the corrected set of 14 paired failure types, and updated the corresponding tables and figures to ensure consistency throughout the manuscript.


[Question 3] Judge Robustness

We thank the reviewer for this suggestion. In the original submission, we evaluated inter-judge agreement on 150 samples (76% three-way agreement). We have now expanded the evaluation to 500 matched control--treatment QA pairs, stratified across all 14 RCA types, and evaluated the same GPT-5.6 Sol predictions using three independent LLM judges.

JudgeMarkdown ControlTreatmentΔ (pp)VLM ControlTreatmentΔ (pp)
GPT-5.4-mini70.2%56.4%13.876.3%69.4%6.9
Claude Sonnet 583.0%68.7%14.388.8%83.5%5.3
Gemini 3.8 Flash82.9%68.4%14.588.6%83.1%5.5

The updated exact agreement among all three judges is 73.5% (1,471 of 2,000 answers). Although Claude and Gemini exhibit more lenient grading behavior, resulting in higher absolute accuracy, all three judges consistently identify similar control–treatment degradation (13.8–14.5 pp for Markdown and 5.3–6.9 pp for VLM). These results confirm that our central findings are robust to evaluator choice. We have updated Appendix A.16 with the expanded analysis.

Summary

We sincerely thank the reviewer for their thoughtful feedback and constructive suggestions. We hope our responses and additional analyses address the key concerns regarding (1) the realism and validity of synthetic failure conditions, (2) cross-model generalizability and ecosystem-level stability, and (3) judge robustness.

We greatly appreciate the reviewer's time and consideration. Thank you again!

Thank you for the response. I plan to maintain my score.

Reviewer_bVcgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SynthFailMD, a synthetic, multi document benchmark generator which aims to recreate failures observed with confidential documents, using synthetic documents. The authors focus on 3 aspects for generating these synthetic examples: document realism, visual realism, and behavioral realism. The authors also introduce a document critique-and-repair method for correcting issues in generated documents.

Critical strengths
  • This paper is well written, easy to follow, and considers an important problem faced by many practitioners.
  • The approach is quite sensible for regenerating failures in synthetic documents, and synthetic samples (at least in A.7) show that the pipeline seems to work quite well.
  • The authors did a good job ablating the question conditioning aspect of the pipeline.
Critical weaknesses
  • I didn't find much discussion which assures that the errors of the agents are the same as the error conditioned failures.
  • The accuracy of the models seems to be quite high (between 50 and 75%), which is surprisingly considering these questions are meant to recreate observed failures?
  • More analysis of why the questions are hard for the agents, and comparison with weaker (ie, luna or terra if we stay in the GPT family) would be desirable to see model size roughly correlates with the accuracy.
Questions for authors
  • Why does DCR decrease the accuracy overall? Would it not increase the accuracy by removing inconsistencies which could get marked as incorrect?
  • How do we know that the model is making the desired mistake from the failure conditioning beyond the ablation of the conditioning?
  • How do we know that the questions/documents are sufficiently diverse during generation?
5 comments

We sincerely appreciate the reviewer’s suggestions. We address your comments and questions:

[Rebuttal Part 1/2]

[Comment 1, Question 2] Validity of error-conditioned failures

To validate whether observed errors correspond to the intended failure mechanisms, we conducted an additional Failure Attribution Analysis in Section 7, Table 4. We identified 1,071 control-correct → treatment-wrong (C→W) transitions on failure-targeted questions. For each transition, an independent LLM judge (GPT-6 Luna) compared model predictions and control–treatment document evidence against the target failure definition to assess whether the observed error was consistent with the intended failure mechanism.

Overall Results: The judge attributed 79.0% (846/1,071) of observed transitions to the intended failure mechanisms, with attribution rates of 84.5% for Canonical Markdown and 67.6% for Direct-PDF VLM.

Evaluation SettingAttributed / C→WAttribution Rate
Canonical Markdown610/72284.5%
Direct-PDF VLM236/34967.6%
Overall846/1,07179.0%

Failure-Specific Attribution: We further examined attribution across individual target failure types.

Target failure typeAttributed / C→WAttribution rate
Handwritten content93/9597.9%
Annotation understanding293/30297.0%
Scanned table recognition179/19094.2%
Merged cell handling32/3688.9%
Scanned text recognition162/21675.0%
Scanned chart recognition2/450.0%
Reading order preservation5/1050.0%
Small/graphical marks78/20138.8%
Cross-page reasoning2/633.3%
Relational reasoning0/60.0%
Column alignment0/50.0%
Overall846/1,07179.0%

Attribution varies across failure types, with particularly strong alignment for handwritten content, annotations, and scanned tables, while lower rates for certain categories highlight opportunities to improve failure conditioning. Together with the failure-specific attribution results (detailed in Section 7, Table 4) and paired intervention analysis (Appendix A.17, Figures 8–9), these findings provide evidence that the injected failure conditions induce errors consistent with their intended mechanisms.


[Comment 2] Overall accuracy relatively high (50–75%)

SynthFailMD aims not only to challenge models but also to systematically characterize their failures. Starting from 27 RCA-derived failure classes, we vary attributes such as document type, number of files, and failure configurations to generate 410 synthetic cases, enabling fine-grained diagnosis of model weaknesses. Furthermore, the reported overall accuracy combines cross-document, variant, and failure-targeted questions, rather than exclusively measuring questions designed to trigger specific failures. Our question-type analysis reveals substantial differences in difficulty, with failure-targeted QA achieving only 14.44% accuracy under Canonical Markdown (Appendix A.11, Table 15). Thus, aggregate accuracy masks the difficulty of targeted failure conditions. Additionally, treatment documents exhibit significantly lower accuracy than their matched controls (Table 3), further demonstrating the benchmark's effectiveness in exposing model weaknesses.

QA TypeGPT-5.6 Sol MarkdownGPT-5.6 Sol VLMGPT-5.5 MarkdownGPT-5.5 VLMGPT-5.4 MarkdownGPT-5.4 VLMOverall MarkdownOverall VLM
Cross-document QA69.04%68.33%69.67%70.30%72.67%66.90%70.46%68.51%
Variant QA69.15%83.12%63.70%83.28%61.95%82.25%64.93%82.88%
Failure-targeted QA15.17%55.81%14.93%57.94%13.22%55.09%14.44%56.28%

[Rebuttal Part 2/2]

[Comment 3] Analysis of question difficulty and weaker models

Analysis of Question Difficulty: Apart from question difficulty in Appendix A.11 and A.12, we conduct an additional failure-type-specific analysis across the 14 matched control–treatment failure types, over three GPT models (GPT-5.6 Sol, GPT-5.5, and GPT-5.4). Under Canonical Markdown, annotation understanding (36.94 pp), scanned table recognition (36.35 pp), and scanned text recognition (33.33 pp) exhibit the largest control-to-treatment degradation. Direct-PDF VLM access mitigates several of these challenges, although annotation understanding remains difficult under both settings. Full control–treatment results with detailed analysis are provided in the new Appendix A.19.

Failure typeMarkdown ControlMarkdown TreatmentΔ (pp)VLM ControlVLM TreatmentΔ (pp)
Scanned chart recognition30.9528.572.3858.1652.725.44
Scanned text recognition68.2334.8933.3373.8855.9517.93
Scanned table recognition77.9741.6236.3585.1280.045.08
Handwritten content55.4339.8115.6260.9559.811.14
Small/Graphical marks61.8033.5228.2870.0462.367.68
Annotation understanding75.3738.4336.9472.4053.2919.11
Relational reasoning69.3471.29−1.9577.1378.10−0.97
Merged cell handling57.2053.253.9467.0661.545.52
Cross-page tables58.7061.35−2.6668.3668.120.24
Column alignment69.0067.371.6380.4279.490.93
Blank/Missing values61.2352.968.2771.1658.6312.53
Cross-page reasoning62.5063.14−0.6472.1265.067.05
Long-context retrieval58.7654.304.4762.8961.861.03
Reading order preservation64.6062.202.4168.0460.487.56

Comparison with Weaker Models: Thanks for the suggestion! We added GPT-5.6 Luna and GPT-5.6 Terra as reviewer suggestions and three additional open-weight models alongside the GPT family (Table 3). GPT models generally achieve higher treatment accuracy; however, performance does not exhibit a strictly monotonic relationship with model size. Appendix A.15 further shows positive correlations across model families in detail.

ModelMarkdown TreatmentVLM Treatment
GPT-5.6 Sol57.62%72.58%
GPT-5.6 Luna53.99%68.04%
GPT-5.6 Terra56.38%67.91%
GPT-5.555.22%73.74%
GPT-5.455.12%71.57%
Kimi K2.552.35%64.68%
Llama 4 Maverick47.64%25.19%
Qwen 3.5 27B50.88%72.17%

[Question 1] Why Does DCR Decrease Overall Accuracy?

DCR evaluates and repairs three aspects of document quality: consistency, coherence, and contradictions (details in Table 10). Most importantly, DCR removes answer leakage in document prose, which allows models to answer correctly without processing the targeted failure conditions. By removing such clues while preserving the intended failures, DCR can reduce artificially inflated QA accuracy. Thus, the lower accuracy observed after DCR (Table 2) is consistent with a more challenging benchmark rather than a deterioration in document quality.


[Question 3] How diverse are generated questions/documents?

We thank the reviewer for raising this concern. We evaluate document diversity through CLIP-based comparisons with real UniDoc documents and Vendi Scores measuring structural, topical, and content diversity (Section 5.3, Figure 3). Tables 11–13 and Figure 6 further demonstrate artifact and question coverage across five domains and 27 RCA types. To assess question-level semantic diversity (Appendix A.10, Table 14), we analyze 3,798 questions using all-mpnet-base-v2 embeddings. Mean pairwise cosine similarity ranges from 0.3120 to 0.3745, with only 0.03%–1.01% of question pairs exceeding a similarity threshold of 0.90 across QA categories. These results provide additional evidence of semantic diversity in the generated questions along with the previous diversity and coverage analysis of documents.

QA TypeQuestionsMean Semantic Similarity ↓High-Similarity Pair Rate (≥0.90) ↓
Cross-document QA1,1660.36330.32%
RCA QA1450.34661.01%
Variant QA1,7960.37450.03%
Visual RCA QA6910.31200.22%

Summary

We thank the reviewer for the detailed feedback and constructive suggestions. We hope our responses and additional analyses address the concerns regarding (1) the validity of injected failure mechanisms, (2) question difficulty and model performance, (3) the impact of DCR on accuracy, and (4) the diversity of generated documents and questions.

We greatly appreciate the reviewer's suggestions. Thank you again!

Thank you for the response. Can the authors please address the comments of the other reviewers as well? In particular, I agree with Reviewer S4rP that better document samples would be very interesting to view!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.