acceptodds
Under review as a conference paper at ICLR 2027

Beyond Tables: Doc2DB-Bench for Relationally Faithful Document-to-Database Construction

Abstract

Practical AI systems increasingly need to turn long, heterogeneous documents into queryable relational databases, not isolated spreadsheets. In domains such as finance, healthcare, education, transportation, and enterprise operations, downstream workflows rely on normalized schemas, entity identities, keys, cross-table relationships, and integrity constraints for analytics, compliance, auditing, and SQL-backed decision making. Existing Document-to-Table benchmarks are insufficient for this setting: flattening evidence into single tables can duplicate entities, obscure many-to-many relationships, create sparse records, and avoid testing whether extracted facts form a valid database instance. This creates an urgent need to evaluate document understanding as database construction rather than field extraction. We introduce Doc2DB-Bench, a benchmark for Document-to-Database construction, containing 203 long-document instances across 42 schemas and seven domain groups, with 117 entity tables, 132 relationship tables, 7,341 rows, and 41,935 cells. Built through a controllable DB-to-Doc synthesis pipeline and organized by a taxonomy of intra-table extraction and inter-table reasoning, the generated documents undergo authenticity verification, proving indistinguishable from real-world references. Doc2DB-Bench thus provides a testbed for reliable, auditable, and relationally faithful LLM-based data systems.

Primary area:
datasets and benchmarks
Keywords:
Information Extraction, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_j8gaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Doc2DB-Bench, a benchmark for evaluating LLMs on the task of constructing relational databases from long documents, going beyond flat-table extraction. The authors design a controllable DB2Doc reverse-synthesis pipeline that starts from existing relational databases (Spider, BIRD), decomposes tuples into atomic evidence, assigns fine-grained capability labels (Intra-Table and Inter-Table taxonomy), and generates long documents with dual validation. The benchmark comprises 203 instances across 42 schemas and 7 domains. Experiments on a wide range of LLMs (open-weight, proprietary, specialized IE systems) show that relation-level extraction remains a major bottleneck, especially for structural reasoning and integrity constraints.

Critical strengths

S1. Document-to-database construction is practically relevant for downstream analytics and SQL-backed systems, yet underexplored in existing benchmarks. The paper clearly motivates why flat-table extraction is insufficient.

S2. The DB2Doc pipeline addresses the scalability bottleneck of manual annotation. The two-pillar taxonomy enables fine-grained diagnosis beyond overall metrics, as demonstrated in Table 4.

Critical weaknesses

W1. The pipeline generates documents via LLMs, but only 50 samples were manually checked, and those checks primarily verified document style rather than task solvability by humans. For example, Identity Resolution requires resolving ambiguous entity mentions across long documents, which is challenging even for human annotators. It is unclear whether humans can produce correct database entries from these documents. This raises questions about the quality of the benchmark.

W2. The evaluation uses a manually-set threshold τ=0.3 (Appendix C.1) to decide whether a predicted row matches a ground-truth row. There is no justification for choosing 0.3, and no sensitivity analysis showing whether model rankings change under different thresholds. If a stricter threshold (e.g., 0.5 or 0.7) alters the relative ordering of models, then the robustness of the reported scores is uncertain.

Questions for authors

Q1. Did the authors verify that human annotators can correctly solve the benchmark instances? If so, what was the human accuracy?

Q2. The matching threshold τ=0.3 (Appendix C.1) seems low. Have the authors conducted sensitivity analysis on this threshold? Would a stricter threshold (e.g., 0.5 or 0.7) change the relative ranking of models?

Q3. Why is the Referential Integrity row entirely blank in Table 4?

Q4. What metric is used in Table 4 and Figs. 4–5? Are the reported scores F1, and are they computed at the cell, row, or relation level?

4 comments

Dear Reviewer j8ga,

We hope this message finds you well. As we approach the end of the discussion period, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,
Authors

Dear Reviewer j8ga,

Apologies for the follow-up, and thank you again for your time and constructive feedback. We have incorporated the clarifications and additional experimental results from our response into the revised manuscript. If you have any remaining questions, we would be happy to address them before the discussion closes.

Thank you again for your time and effort in reviewing our paper.

Best regards,
The Authors

Thanks for the additional analysis and clarification. My concerns have been addressed. I will raise my rating from 2 to 3. One minor point: the authors may explain in the revised paper why the "Null Relation Extraction" row in Table 5 is blank.

Dear Reviewer j8ga,

Thank you for carefully considering our response. We are glad that the clarifications addressed your primary concerns and appreciate your constructive feedback.

We have incorporated the explanation for the blank “Null Relation Extraction” row in Table 5, along with the additional experimental results, into the revised manuscript.

Best regards,
The Authors

Reviewer_zSQcOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a benchmark evaluating whether agents can construct database from the documents which is an important problem in real-world. Extensive exepriments on the benchmark demonstrate that existing models still fall short in the problem and the authors provides some insightful findings from the reults.

Critical strengths
  1. Document-to-Database is an important task in real-word data processing, the benchmark provides a good chance for evaluating agents on such problem.
  2. The benchmark is constructed based on real-world databases and the proposed DB-to-Doc synthesis pipeline guarantees the verification effectiveness.
  3. Extensive experiments are conducted and results are further analyzed to give insights on the shortcommings of existing models.
Critical weaknesses
  1. Only cell-level F1 is provided, more metrics can help us better understand and compare the performance of different models.
  2. Schema is provided to agents instead of open-end database construction, which is more realistic.
  3. Lack of case studies comparing different model predictions to illustrate the actual failure modes of LLMs.
Questions for authors
  1. Why provide the agents with schema instead of letting models to generate it.
  2. I wonder if a model achieve high cell-level F1 while still producing an invalid relational database.
  3. Can a model achieve high cell-level F1 while still producing an invalid relational database?
Reviewer_tRibOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that existing document-to-table extraction aim to populate a single flattened table, and ignores challenges that arise when the expected output is a coherent relational database. Identity resolution, cross-table relationships, and integrity constraints are important aspects of database construction that existing flat table extraction may ignore. It proposes a Doc2DB-Bench, a benchmark for evaluating LLMs on constructing relational databases from long documents. The benchmark is constructed using a reverse synthesis pipeline that serializes evidence from BIRD and Spider benchmarks into documents.

Critical strengths
  1. The paper highlights a conceptual gap between reliably extracting individual facts and constructing a relational database.
  2. Simple, controlled benchmark generation pipeline. A taxonomy of intra- and inter-table capabilities.
  3. Benchmark with capability taxonomy enables a finer-grained analysis of failure modes in the task.
Critical weaknesses
  1. My main concern is missing a related body of work on open-web research or broad entity collection (e.g. BrowseComp, SODIUM, WideSearch, DeepWideSearch, WideSeekBench and others). With this space becoming increasingly crowded, it is critical to distinguish the specific challenges of relational construction task from the broader problem of assembling structured information. For example, which capabilities from entity resolution, relationship reconstruction, multi-hop composition, and semantic integrity are not covered by prior benchmarks. The contribution should be grounded in these specific gaps rather than primarily in the use of long documents or multi-table outputs. The paper could be stronger if it articulated a more specific research question about the challenges of relational database construction and used controlled experiments to investigate the underlying failure modes.
  2. There are known quality issues with BIRD and Spider benchmarks (related to ground truths in these datasets).
  3. Insufficient statistics about the resulting benchmark, especially related to diversity.
  4. The paper could benefit from more controlled comparisons that show whether specific capabilities account for the performance gap, or if the errors are mostly due to the general difficulty of extracting structured information from long documents.
  5. The paper's format/styling seems off, especially spacing.
Questions for authors

What specific evaluation gaps does Doc2DB-Bench address that existing benchmarks for database construction and wide information gathering do not?

4 comments

Dear Reviewer tRib,

We hope this message finds you well. As we approach the end of the discussion period, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,
Authors

Dear Reviewer tRib,

Apologies for the follow-up, and thank you again for your time and constructive feedback. We have incorporated the clarifications and additional experimental results from our response into the revised manuscript. If you have any remaining questions, we would be happy to address them before the discussion closes.

Best regards,
The Authors

Thank you for the detailed responses and clarification. The authors have adequately addressed my concerns about the positioning of the work. I am willing to raise my rating to Weak Accept.

Dear Reviewer tRib,

Thank you for carefully considering our response. We are glad that the clarifications addressed your primary concerns and appreciate your constructive feedback.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.