- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces RELIC, a benchmark of about 16,000 multi-table instances with six types of data integrity violations. It tests whether LLMs detect these problems and disclose them in their answers. Across 20 models, most corrupted inputs are answered without warning, even when the reasoning trace identifies the problem. Prompt interventions help with some violation types but do not consistently reduce these silent failures.
Critical strengths
S1. The authors curated a new benchmark datastes.
S2. The dataset covers six types of integrity violations and 20 models were examined.
S3. Interesting results were revealed.
Critical weaknesses
W1. SFR treats any answer without abstention as a silent failure in the base setting, even when the model correctly repairs duplicated data. Table 2 also shows that abstention depends strongly on the prompt and violation type. For QwQ-32B, listing the violations explicitly raises the abstention shift on DUP from 60.1 to 76.3 percentage points, but lowers it on ORPHAN-FK from 22.2 to 6.8. These changes do not establish whether the answers become more accurate or reliable. The evaluation should distinguish wrong answers without warnings, correct answers after repair, and justified abstention. Otherwise, a lower SFR may simply reflect more refusals rather than better handling of corrupted data.
W2. The paper needs to explain when these integrity checks should be handled by an LLM. In databases with enforced primary-key and foreign-key constraints, many of the tested violations would already be prevented, and standard database checks can detect them directly. Even when integrity violations occur, why not apply correction techniques to fix problems first? Why do we need to reason against the corrupted data? What practical benefit the proposed evaluation offers, over checking the data before passing it to an LLM?
W3. Each clean instance has 12 corrupted versions that share the same underlying tables and query. The paper should clarify whether every instance is evaluated in a separate conversation without shared history. If related instances are tested in the same conversation, the model could use the clean tables or earlier answers to handle later corrupted inputs.
Questions for authors
Please address weaknesses W1-W3.
6 comments
Response to Reviewer pUA7 (2/3)
We believe that a violation should be disclosed to the user before it is corrected, because correction decides how the violation is handled without the user's knowledge. Correction requires two steps: (i) knowing that a violation exists and (ii) deciding how to handle it. Step (ii) depends on what the question asks. For example, if some orders in a question about "total order amount per customer" have customer IDs that do not appear in the customer table, whether to exclude those orders from the total, report them separately, or decline to answer is a decision for the user. Correction techniques and SQL make this choice without asking the user and apply one fixed handling: deleting orphan rows removes their amounts from the total, and merging conflicting duplicate rows keeps only one of the two values; an inner join drops unmatched rows, and aggregate functions skip NULL values (SUM(30, NULL) returns 30 [5]). In each case the result carries no indication that the data were changed, so the user cannot tell whether the answer reflects the intended handling. In the setting evaluated in the paper, the user provides the question and the tables directly to the model and receives only the answer, so the model is the only party that can inform the user of the violation. SFR measures whether it does so.
If the problem with correction techniques is that they apply one fixed handling, one alternative is to let the LLM inspect the tables itself with a code-execution tool and handle the violation according to the context of the question. To test whether this works, we gave GLM-5 and Kimi-K2 a code-execution tool and evaluated how they handle corrupted tables, using the output format with a reporting field (issues) in which the model lists the problems it found alongside its answer. Each response was paired with the tool-free response on the same instance (Table 2).
Table 2. Results with and without a code-execution tool (%, output format with the reporting field; tool-free = no tool, tool-use = with the code-execution tool)
| GLM-5 | Kimi-K2 | |||
|---|---|---|---|---|
| tool-free | tool-use | tool-free | tool-use | |
| Clean accuracy | 76.7 | 100.0 | 66.7 | 96.7 |
| Disclosure (abstention or report of the injected violation type) | 45.1 | 43.3 (McNemar p = .67) | 58.9 | 52.9 (p = .09) |
| False alarms on clean inputs | 6.7 | 0.0 | 16.7 | 6.7 |
| Tool-call rate (corrupted instances) | – | 100.0 | – | 78.3 |
| Calls containing an integrity check | – | 18.6 | – | 41.4 |
| Check output exposed the violation, model did not report it | – | 15.6 | – | 34.1 |
| Answer equals the result of executing the query on the corrupted tables as they are | – | 92.7 | – | 88.8 |
Even with a code-execution tool, the models mostly did not check for violations, and disclosure did not increase. The tool increased clean accuracy by 23.3 and 30.0 points, but disclosure on corrupted inputs (abstention or a report of the injected violation type) did not change significantly (45.1% → 43.3% and 58.9% → 52.9%). Only 18.6% and 41.4% of the tool calls contained an integrity check, and 92.7% and 88.8% of the answers equal the result of executing the query on the corrupted tables as they are. Thus, even with a tool, the models mostly did not perform step (i).
We see the practical benefit of RELIC in that it measures whether the model discloses a violation when tables are provided without a prior check, and it identifies, for each violation type, which capability of the model needs to improve. Duplicated keys are reported in 72–99% of cases in the majority of models once a reporting field is available, which indicates that models detect them and only need a means of disclosure. Orphaned keys, by contrast, are reported in only 3–36% of cases in all models even with the reporting field, and the tool did not increase disclosure in the experiment above; this suggests that the ability to detect violations by comparing keys across tables needs to improve (Sec. 5.4–5.6). For inputs where constraint checking is not available, this diagnosis offers a starting point for improving how models handle violations.
[1] Narayan et al. Can Foundation Models Wrangle Your Data? VLDB 2022.
[2] Coelho, Aillos, Pilot, and Valeev. On the Quality of Relational Database Schemas in Open-source Software. International Journal on Advances in Software 4(3&4), 2011.
[3] Zhang, Hadjieleftheriou, Ooi, Procopiuc, and Srivastava. On Multi-Column Foreign Key Discovery. PVLDB 3(1), 2010.
[4] SQLite. Foreign Key Support. SQLite documentation.
[5] Toussaint, Guagliardo, Libkin, and Sequeda. Troubles with Nulls, Views from the Users. VLDB 2022.
Response to Reviewer pUA7 (3/3)
W3. Each clean instance has 12 corrupted versions that share the same underlying tables and query. The paper should clarify whether every instance is evaluated in a separate conversation without shared history. If related instances are tested in the same conversation, the model could use the clean tables or earlier answers to handle later corrupted inputs.
Thank you for raising this point. We confirm that every instance is evaluated in a single, independent API call with no conversation history. Each call consists of one system prompt, one user prompt containing the query and the two tables, and one response (Sec. 4.1, Appendix C). In no instance can the model see the clean tables, another corrupted version of the same problem, or a previous response. As suggested, the revision states this explicitly in Sec. 4.1 and Appendix F.
Response to Reviewer pUA7 (1/3)
We are grateful to the reviewer for the careful reading and helpful suggestions. Our responses to each comment are given below.
Thank you for this comment. Following this suggestion, we explicitly classified every corrupted response into one of four categories: (a) a wrong answer without warning, (b) an answer that matches the original gold answer without warning, (c) an abstention, and (d) an answer with a report of the violation. Responses in (b) are counted as silent failures even though the answer is correct, because they do not disclose the violation: the user needs to know that a violation was present in order to decide whether to rely on the answer. We then applied this classification to the results of four conditions (base, informed, explicit, and issues; Table 1; Appendix E.3). Informed and explicit are prompts that instruct the model to verify integrity; issues keeps the base prompt and adds a reporting field to the output.
Table 1. Four-way classification of corrupted responses and abstention rate on clean inputs per condition (%; macro-average over the nine models of Table 2 of the paper, range across models in parentheses)
This classification shows that the reduction in SFR is not due to a general increase in refusals. First, the verification instructions increase the abstention rate on corrupted inputs from 4.0% to 14.5% and 17.7%, whereas the rate on clean inputs increases by only 0.6 and 0.9 points. Second, we agree with the reviewer that more abstention does not by itself imply more reliable answers: of the 13.7 points of additional abstention under explicit, 6.8 points replace wrong answers without warning (78.7% → 71.9%) and 6.8 points replace answers that matched the original gold (17.3% → 10.5%). Third, the reporting field gives the largest reduction in wrong answers without warning (78.7% → 60.8%), and 27.2% of the responses become answers with a report.
Thank you for these questions. In response, we clarify the setting in which an LLM must handle integrity violations (Sec. 1), and we added an experiment in which the model can check the tables with a code-execution tool (Sec. 5.6, Appendix E.7).
Database constraint checking is often not available for the inputs that RELIC targets. In the setting evaluated in the paper, the LLM receives the query together with the tables and returns the answer. The input is then a file exported as CSV or Markdown, which carries no key-constraint declarations [1]. Foreign keys are also frequently left undeclared in the source database [2][3]: in a survey of the schemas of 512 open-source software projects, 88% of the MySQL schemas and 55% of the PostgreSQL schemas declare no foreign key [2]. SQLite does not check foreign keys in its default configuration [4], and 63.6% of the BIRD dev databases contain rows that violate a foreign key declared in their own schema (Appendix A). In this setting, violations can therefore reach the model without having been prevented or detected.
(continued in the next comment)