- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Doc2DB-Bench, a benchmark for evaluating LLMs on the task of constructing relational databases from long documents, going beyond flat-table extraction. The authors design a controllable DB2Doc reverse-synthesis pipeline that starts from existing relational databases (Spider, BIRD), decomposes tuples into atomic evidence, assigns fine-grained capability labels (Intra-Table and Inter-Table taxonomy), and generates long documents with dual validation. The benchmark comprises 203 instances across 42 schemas and 7 domains. Experiments on a wide range of LLMs (open-weight, proprietary, specialized IE systems) show that relation-level extraction remains a major bottleneck, especially for structural reasoning and integrity constraints.
Critical strengths
S1. Document-to-database construction is practically relevant for downstream analytics and SQL-backed systems, yet underexplored in existing benchmarks. The paper clearly motivates why flat-table extraction is insufficient.
S2. The DB2Doc pipeline addresses the scalability bottleneck of manual annotation. The two-pillar taxonomy enables fine-grained diagnosis beyond overall metrics, as demonstrated in Table 4.
Critical weaknesses
W1. The pipeline generates documents via LLMs, but only 50 samples were manually checked, and those checks primarily verified document style rather than task solvability by humans. For example, Identity Resolution requires resolving ambiguous entity mentions across long documents, which is challenging even for human annotators. It is unclear whether humans can produce correct database entries from these documents. This raises questions about the quality of the benchmark.
W2. The evaluation uses a manually-set threshold τ=0.3 (Appendix C.1) to decide whether a predicted row matches a ground-truth row. There is no justification for choosing 0.3, and no sensitivity analysis showing whether model rankings change under different thresholds. If a stricter threshold (e.g., 0.5 or 0.7) alters the relative ordering of models, then the robustness of the reported scores is uncertain.
Questions for authors
Q1. Did the authors verify that human annotators can correctly solve the benchmark instances? If so, what was the human accuracy?
Q2. The matching threshold τ=0.3 (Appendix C.1) seems low. Have the authors conducted sensitivity analysis on this threshold? Would a stricter threshold (e.g., 0.5 or 0.7) change the relative ranking of models?
Q3. Why is the Referential Integrity row entirely blank in Table 4?
Q4. What metric is used in Table 4 and Figs. 4–5? Are the reported scores F1, and are they computed at the cell, row, or relation level?
4 comments
Dear Reviewer j8ga,
Apologies for the follow-up, and thank you again for your time and constructive feedback. We have incorporated the clarifications and additional experimental results from our response into the revised manuscript. If you have any remaining questions, we would be happy to address them before the discussion closes.
Thank you again for your time and effort in reviewing our paper.
Best regards,
The Authors
Thanks for the additional analysis and clarification. My concerns have been addressed. I will raise my rating from 2 to 3. One minor point: the authors may explain in the revised paper why the "Null Relation Extraction" row in Table 5 is blank.
Dear Reviewer j8ga,
Thank you for carefully considering our response. We are glad that the clarifications addressed your primary concerns and appreciate your constructive feedback.
We have incorporated the explanation for the blank “Null Relation Extraction” row in Table 5, along with the additional experimental results, into the revised manuscript.
Best regards,
The Authors
Dear Reviewer j8ga,
We hope this message finds you well. As we approach the end of the discussion period, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper.
Best regards,
Authors