acceptodds
Under review as a conference paper at ICLR 2027

Reasoning over Broken Joins: How LLMs Handle Integrity Violations in Multi-table Inputs

Abstract

Large language models are increasingly used to query multi-table relational data. The correctness of such queries depends on integrity constraints such as key uniqueness and referential integrity, yet these constraints are frequently violated in practice. On such corrupted inputs, a model should detect the violation and disclose it rather than answer silently. Existing evaluations on such corrupted inputs score only the final output. They therefore cannot distinguish a model that misses a violation from one that detects it but answers without disclosing it. We introduce RELIC, a dataset of approximately 16,000 multi-table instances in which six violation types are injected at controlled severities into integrity-verified synthetic table pairs. We define the silent failure rate (SFR) as the fraction of corrupted instances that a model answers without disclosing the violation. We further decompose silent failures by whether the model detects the violation in its reasoning trace. SFR exceeds 84% for all 20 models and 97% for all three frontier models despite clean accuracy of up to 98.3%. Detection increases with model capability and reasoning budget while disclosure does not. Prompt-level interventions elicit disclosure of duplicated primary keys from reasoning models but leave most orphaned foreign keys undisclosed. Model scaling and prompt-level interventions therefore fail to mitigate silent failures consistently across violation types. RELIC enables fine-grained attribution of silent failures to detection or disclosure for each violation type, facilitating the development of LLMs that handle integrity violations reliably.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
data integrity, integrity constraints, relational tables, abstention

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pUA7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces RELIC, a benchmark of about 16,000 multi-table instances with six types of data integrity violations. It tests whether LLMs detect these problems and disclose them in their answers. Across 20 models, most corrupted inputs are answered without warning, even when the reasoning trace identifies the problem. Prompt interventions help with some violation types but do not consistently reduce these silent failures.

Critical strengths

S1. The authors curated a new benchmark datastes.

S2. The dataset covers six types of integrity violations and 20 models were examined.

S3. Interesting results were revealed.

Critical weaknesses

W1. SFR treats any answer without abstention as a silent failure in the base setting, even when the model correctly repairs duplicated data. Table 2 also shows that abstention depends strongly on the prompt and violation type. For QwQ-32B, listing the violations explicitly raises the abstention shift on DUP from 60.1 to 76.3 percentage points, but lowers it on ORPHAN-FK from 22.2 to 6.8. These changes do not establish whether the answers become more accurate or reliable. The evaluation should distinguish wrong answers without warnings, correct answers after repair, and justified abstention. Otherwise, a lower SFR may simply reflect more refusals rather than better handling of corrupted data.

W2. The paper needs to explain when these integrity checks should be handled by an LLM. In databases with enforced primary-key and foreign-key constraints, many of the tested violations would already be prevented, and standard database checks can detect them directly. Even when integrity violations occur, why not apply correction techniques to fix problems first? Why do we need to reason against the corrupted data? What practical benefit the proposed evaluation offers, over checking the data before passing it to an LLM?

W3. Each clean instance has 12 corrupted versions that share the same underlying tables and query. The paper should clarify whether every instance is evaluated in a separate conversation without shared history. If related instances are tested in the same conversation, the model could use the clean tables or earlier answers to handle later corrupted inputs.

Questions for authors

Please address weaknesses W1-W3.

6 comments
AuthorsOpenReview ↗

Response to Reviewer pUA7 (1/3)

We are grateful to the reviewer for the careful reading and helpful suggestions. Our responses to each comment are given below.

W1. SFR treats any answer without abstention as a silent failure in the base setting, even when the model correctly repairs duplicated data. Table 2 also shows that abstention depends strongly on the prompt and violation type. For QwQ-32B, listing the violations explicitly raises the abstention shift on DUP from 60.1 to 76.3 percentage points, but lowers it on ORPHAN-FK from 22.2 to 6.8. These changes do not establish whether the answers become more accurate or reliable. The evaluation should distinguish wrong answers without warnings, correct answers after repair, and justified abstention. Otherwise, a lower SFR may simply reflect more refusals rather than better handling of corrupted data.

Thank you for this comment. Following this suggestion, we explicitly classified every corrupted response into one of four categories: (a) a wrong answer without warning, (b) an answer that matches the original gold answer without warning, (c) an abstention, and (d) an answer with a report of the violation. Responses in (b) are counted as silent failures even though the answer is correct, because they do not disclose the violation: the user needs to know that a violation was present in order to decide whether to rely on the answer. We then applied this classification to the results of four conditions (base, informed, explicit, and issues; Table 1; Appendix E.3). Informed and explicit are prompts that instruct the model to verify integrity; issues keeps the base prompt and adds a reporting field to the output.

Table 1. Four-way classification of corrupted responses and abstention rate on clean inputs per condition (%; macro-average over the nine models of Table 2 of the paper, range across models in parentheses)

ConditionWrong, no warningMatches original gold, no warningAbstentionAnswer with a report of the violationAbstention on clean inputs
base78.7 (69.9–97.1)17.3 (2.4–29.6)4.0 (0.6–9.5)–1.4 (0.0–5.0)
informed72.6 (60.7–96.5)12.8 (2.6–28.6)14.5 (0.8–32.4)–2.0 (0.0–7.7)
explicit71.9 (55.3–96.3)10.5 (3.1–20.7)17.7 (0.6–34.0)–2.3 (0.0–5.9)
issues60.8 (36.9–87.4)7.3 (2.0–11.5)4.7 (0.5–12.4)27.2 (7.9–55.3)1.8 (0.0–7.5)

This classification shows that the reduction in SFR is not due to a general increase in refusals. First, the verification instructions increase the abstention rate on corrupted inputs from 4.0% to 14.5% and 17.7%, whereas the rate on clean inputs increases by only 0.6 and 0.9 points. Second, we agree with the reviewer that more abstention does not by itself imply more reliable answers: of the 13.7 points of additional abstention under explicit, 6.8 points replace wrong answers without warning (78.7% → 71.9%) and 6.8 points replace answers that matched the original gold (17.3% → 10.5%). Third, the reporting field gives the largest reduction in wrong answers without warning (78.7% → 60.8%), and 27.2% of the responses become answers with a report.

W2. The paper needs to explain when these integrity checks should be handled by an LLM. In databases with enforced primary-key and foreign-key constraints, many of the tested violations would already be prevented, and standard database checks can detect them directly. Even when integrity violations occur, why not apply correction techniques to fix problems first? Why do we need to reason against the corrupted data? What practical benefit the proposed evaluation offers, over checking the data before passing it to an LLM?

Thank you for these questions. In response, we clarify the setting in which an LLM must handle integrity violations (Sec. 1), and we added an experiment in which the model can check the tables with a code-execution tool (Sec. 5.6, Appendix E.7).

Database constraint checking is often not available for the inputs that RELIC targets. In the setting evaluated in the paper, the LLM receives the query together with the tables and returns the answer. The input is then a file exported as CSV or Markdown, which carries no key-constraint declarations [1]. Foreign keys are also frequently left undeclared in the source database [2][3]: in a survey of the schemas of 512 open-source software projects, 88% of the MySQL schemas and 55% of the PostgreSQL schemas declare no foreign key [2]. SQLite does not check foreign keys in its default configuration [4], and 63.6% of the BIRD dev databases contain rows that violate a foreign key declared in their own schema (Appendix A). In this setting, violations can therefore reach the model without having been prevented or detected.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer pUA7 (2/3)

We believe that a violation should be disclosed to the user before it is corrected, because correction decides how the violation is handled without the user's knowledge. Correction requires two steps: (i) knowing that a violation exists and (ii) deciding how to handle it. Step (ii) depends on what the question asks. For example, if some orders in a question about "total order amount per customer" have customer IDs that do not appear in the customer table, whether to exclude those orders from the total, report them separately, or decline to answer is a decision for the user. Correction techniques and SQL make this choice without asking the user and apply one fixed handling: deleting orphan rows removes their amounts from the total, and merging conflicting duplicate rows keeps only one of the two values; an inner join drops unmatched rows, and aggregate functions skip NULL values (SUM(30, NULL) returns 30 [5]). In each case the result carries no indication that the data were changed, so the user cannot tell whether the answer reflects the intended handling. In the setting evaluated in the paper, the user provides the question and the tables directly to the model and receives only the answer, so the model is the only party that can inform the user of the violation. SFR measures whether it does so.

If the problem with correction techniques is that they apply one fixed handling, one alternative is to let the LLM inspect the tables itself with a code-execution tool and handle the violation according to the context of the question. To test whether this works, we gave GLM-5 and Kimi-K2 a code-execution tool and evaluated how they handle corrupted tables, using the output format with a reporting field (issues) in which the model lists the problems it found alongside its answer. Each response was paired with the tool-free response on the same instance (Table 2).

Table 2. Results with and without a code-execution tool (%, output format with the reporting field; tool-free = no tool, tool-use = with the code-execution tool)

GLM-5Kimi-K2
tool-freetool-usetool-freetool-use
Clean accuracy76.7100.066.796.7
Disclosure (abstention or report of the injected violation type)45.143.3 (McNemar p = .67)58.952.9 (p = .09)
False alarms on clean inputs6.70.016.76.7
Tool-call rate (corrupted instances)–100.0–78.3
Calls containing an integrity check–18.6–41.4
Check output exposed the violation, model did not report it–15.6–34.1
Answer equals the result of executing the query on the corrupted tables as they are–92.7–88.8

Even with a code-execution tool, the models mostly did not check for violations, and disclosure did not increase. The tool increased clean accuracy by 23.3 and 30.0 points, but disclosure on corrupted inputs (abstention or a report of the injected violation type) did not change significantly (45.1% → 43.3% and 58.9% → 52.9%). Only 18.6% and 41.4% of the tool calls contained an integrity check, and 92.7% and 88.8% of the answers equal the result of executing the query on the corrupted tables as they are. Thus, even with a tool, the models mostly did not perform step (i).

We see the practical benefit of RELIC in that it measures whether the model discloses a violation when tables are provided without a prior check, and it identifies, for each violation type, which capability of the model needs to improve. Duplicated keys are reported in 72–99% of cases in the majority of models once a reporting field is available, which indicates that models detect them and only need a means of disclosure. Orphaned keys, by contrast, are reported in only 3–36% of cases in all models even with the reporting field, and the tool did not increase disclosure in the experiment above; this suggests that the ability to detect violations by comparing keys across tables needs to improve (Sec. 5.4–5.6). For inputs where constraint checking is not available, this diagnosis offers a starting point for improving how models handle violations.


[1] Narayan et al. Can Foundation Models Wrangle Your Data? VLDB 2022.

[2] Coelho, Aillos, Pilot, and Valeev. On the Quality of Relational Database Schemas in Open-source Software. International Journal on Advances in Software 4(3&4), 2011.

[3] Zhang, Hadjieleftheriou, Ooi, Procopiuc, and Srivastava. On Multi-Column Foreign Key Discovery. PVLDB 3(1), 2010.

[4] SQLite. Foreign Key Support. SQLite documentation.

[5] Toussaint, Guagliardo, Libkin, and Sequeda. Troubles with Nulls, Views from the Users. VLDB 2022.

AuthorsOpenReview ↗

Response to Reviewer pUA7 (3/3)

W3. Each clean instance has 12 corrupted versions that share the same underlying tables and query. The paper should clarify whether every instance is evaluated in a separate conversation without shared history. If related instances are tested in the same conversation, the model could use the clean tables or earlier answers to handle later corrupted inputs.

Thank you for raising this point. We confirm that every instance is evaluated in a single, independent API call with no conversation history. Each call consists of one system prompt, one user prompt containing the query and the two tables, and one response (Sec. 4.1, Appendix C). In no instance can the model see the clean tables, another corrupted version of the same problem, or a previous response. As suggested, the revision states this explicitly in Sec. 4.1 and Appendix F.

Reviewer_SdBsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how LLMs handle errors in relational tables, particularly whether they recognize these errors and alert users before answering. The authors introduce RELIC, a benchmark of 16,120 instances with six types of integrity violations, and propose the Silent Failure Rate (SFR) to measure how often models answer without reporting them. They also distinguish between models that fail to detect an error and those that detect it but do not disclose it. Across 20 models, SFR exceeds 84%, including over 97% for frontier models. More reasoning improves error detection but does little to improve disclosure. Prompting helps models identify and report duplicate keys, but is less effective for broken foreign-key references. Overall, the paper shows that recognizing errors does not necessarily mean models will communicate them to users.

Critical strengths
  • The paper addresses an important problem. Most evaluations focus on whether LLMs produce correct answers, whereas this work examines whether models recognize errors in the underlying data and communicate them to users
  • The distinction between failing to detect an error and detecting it without reporting it is useful. It provides a more nuanced understanding of model failures than answer accuracy alone
  • RELIC is systematically constructed with six violation types and two severity levels. Using synthetic tables based on real database schemas allows the authors to control the injected errors and evaluate their effects more precisely
  • The evaluation is extensive, covering 20 models with different capabilities and reasoning settings. The experiments on reasoning effort, corruption severity, table size, and prompting help establish how robust the findings are across conditions.
  • The results reveal interesting differences across violation types. In particular, models respond better to prompting for duplicate keys than for broken foreign key references, suggesting that different errors may require different approaches to detection and disclosure
Critical weaknesses
  • My main concern is with how silent failure is measured. In the main experiments models cannot report an error while providing an answer, so SFR is essentially the rate at which they choose not to abstain. I'm not sure this is sufficient to support the paper's central distinction between detecting an error and failing to disclose it. Section 5.5 even acknowledges that silence around duplicate keys largely reflects the absence of a reporting field. Why isn't the reporting-field setup used throughout the evaluation?
  • I'm also not entirely convinced by how the authors determine whether a model detected an error. They search reasoning traces for certain keywords but these can appear even when the model has not identified the actual problem. For example, saying that a table has no duplicates can still count as detection. The reported validation results are strong (100% precision and 92.1% recall) but they come from only 100 traces, and its unclear which models were included. I would be interested in seeing how often the same detector flags clean inputs, especially since longer reasoning traces have more opportunities to contain these keywords
  • Similarly I'm not sure abstention is always the right response to an integrity violation. SQL normally ignores NULL values during aggregation and the paper notes that CONFLICT-DUP cases can still produce the correct answer. The authors examine silent repairs for duplicated rows but do not report answer correctness consistently across all violation types… I think this distinction matters when interpreting how serious the reported failures actually are
  • Another concern is that many of the evaluated models struggle with the task even when the tables are clean. In Table 1, 14 of the 20 models achieve 25% accuracy or less. If a model cannot reliably answer questions over clean tables, it is difficult to draw meaningful conclusions about its ability to recognize and handle corrupted data… I think the authors should prioritize on models that can perform the underlying task reasonably well
  • I wonder how well the results generalize beyond the benchmark. The experiments use synthetic tables with relatively simple two-table joins and aggregation queries. Real databases can involve more complex relationships and multiple interacting errors and it is unclear whether the same detection and disclosure patterns would emerge in those settings
  • Some of the discussion goes beyond what the experiments establish. For example, the suggestion that post-training incentives encourage models to answer rather than disclose errors is plausible, but not directly tested. The evidence supports a behavioral observation rather than a causal explanation
  • Finally, asking models to explicitly report errors introduces another problem: they incorrectly flag 7.7–20% of clean inputs. That seems substantial and these false alarms could be costly in practice. The paper also does not establish whether the gains in disclosure outweigh them
Questions for authors

Please see my comments above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer SdBs (1/5)

We are grateful to the reviewer for the careful reading and helpful suggestions. Below, we quote each comment and give our response.

W1. My main concern is with how silent failure is measured. In the main experiments models cannot report an error while providing an answer, so SFR is essentially the rate at which they choose not to abstain. I'm not sure this is sufficient to support the paper's central distinction between detecting an error and failing to disclose it. Section 5.5 even acknowledges that silence around duplicate keys largely reflects the absence of a reporting field. Why isn't the reporting-field setup used throughout the evaluation?

Thank you for raising this point. To address this concern, we extended the reporting-field condition from three to eleven models (Sec. 5.5, Appendix E.4), so that failures to detect a violation can be separated from failures to disclose it.

We kept the reporting field out of the default condition because the field itself acts as a request to look for violations. The reporting field (issues) is an output field in which the model lists the problems it found alongside its answer. RELIC measures whether a model informs the user of a violation when it has not been asked to look for one. With the reporting field as the default, the benchmark would measure whether the model can report violations on request. With the field, models also report a problem on 7.7–20% of clean tables (false alarms; Figure 6c). Standard benchmarks for table QA, text-to-SQL, and data-analysis agents return a single answer and have no field for data-quality issues [1][2][3][4][5], and base corresponds to this format. Abstention is the only means of disclosing a violation in this format, so SFR equals 1 − abstention rate under base by design. The base prompt states that the tables may be corrupted and permits abstention (Sec. 4.1), which makes base more favourable to the model than real use.

We agree that SFR under base alone cannot establish the distinction between detection and non-disclosure; the distinction rests on the comparison of base with the reporting-field condition. The reporting-field condition does not replace base; it is an auxiliary condition that diagnoses the cause of the silence observed under base. With the reporting field, duplicated keys are reported in 72–99% of cases in the majority of the eleven models. These models can identify duplicated keys, so their silence under base is a failure to disclose a violation that they can identify; the statement in Sec. 5.5 refers to this result. Orphaned keys are reported in only 3–36% of cases in all models even with the reporting field, which is a detection failure.

With the reporting field, SFR decreases by 6–59 points in all eleven models but remains at 41–80% (Table 1). Thus, at most these 6–59 points are due to the absence of a means of reporting a violation under base; the remaining 41–80% occurs even when such a means is available. Table 1 also reports the split of silent failures in each condition into G (detected but not disclosed) and B (not detected) under the grounded detector (see our response to W2).

Table 1. SFR and detection decomposition with and without the reporting field (% of corrupted inputs; G and B under the grounded detector; base = without the reporting field, issues = with the reporting field; G = detected but not disclosed, B = not detected)

Modelbaseissues
SFRGBSFRGB
GLM-599.246.241.854.921.632.4
Kimi-K299.766.028.041.120.719.3
gpt-oss-120B93.247.336.452.325.725.7
gpt-oss-20B92.336.344.572.433.435.7
Qwen3-32B93.948.040.758.525.830.8
QwQ-32B92.453.834.564.037.824.0
R1-Distill-70B86.032.753.579.631.744.0
Qwen3-8B92.423.762.276.317.556.9
Llama-3.3-70B88.723.464.965.810.254.0
gemma-4-31B96.823.562.666.819.047.8
R1-Distill-7B95.34.988.164.15.158.9

[1] Pasupat and Liang. Compositional Semantic Parsing on Semi-Structured Tables. ACL 2015.

[2] Yu et al. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. EMNLP 2018.

[3] Li et al. Can LLM Already Serve as a Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. NeurIPS 2023.

[4] Hu et al. InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks. ICML 2024.

[5] Wu et al. TableBench: A Comprehensive and Complex Benchmark for Table Question Answering. AAAI 2025.

AuthorsOpenReview ↗

Response to Reviewer SdBs (2/5)

W2. I'm also not entirely convinced by how the authors determine whether a model detected an error. They search reasoning traces for certain keywords but these can appear even when the model has not identified the actual problem. For example, saying that a table has no duplicates can still count as detection. The reported validation results are strong (100% precision and 92.1% recall) but they come from only 100 traces, and it's unclear which models were included. I would be interested in seeing how often the same detector flags clean inputs, especially since longer reasoning traces have more opportunities to contain these keywords.

Thank you for this detailed comment. To address this concern, we measured how often the detector fires on clean traces, replaced it with a stricter grounded detector, and added controls for negation and trace length.

As the reviewer anticipated, the original detector also fires on clean traces, because it only tests for the presence of detection keywords. The base prompt mentions possible corruption, so models describe the checks they perform even on clean inputs, and the original detector fires on these descriptions. The validation set of 100 traces consists of 25 corrupted traces under base from each of GLM-5, Kimi-K2, Qwen3-32B, and gpt-oss-120B; this composition is added to Appendix D.1.

The revision therefore re-computes G and B with a grounded detector that also requires the injected key value, and uses it as the primary measure (Sec. 4.2). The grounded detector fires only when a detection keyword and the injected key value occur in the same sentence of the trace. For clean traces, which contain no injected value, the join key values and column names of the same problem are used instead. Details of the grounded detector are given in Appendix D.2.

Table 2. Grounded detector: clean firing rate and retained G (clean denominator = traces that can be scored; retained G = grounded G ÷ G of the original detector)

Original detectorGrounded
Clean firing rate, all runs (range, median).06–.97 (.35).00–.28 (.04)
Clean firing rate, issues condition (11 runs).73–.96.01–.14
Retained G, reasoning models (16 runs)1.56–.77 (median .72)
Retained G, gemma-4-31B1.49–.60
Retained G, R1-Distill-7B1.18–.31

The firing rate on clean traces decreases from a median of .35 to .04 and from a maximum of .97 to .28 (Table 2). In contrast, G (detected but not disclosed) on corrupted traces largely remains under the stricter criterion: G computed with the grounded detector is 56–77% of G computed with the original detector for reasoning models (median 72%) and 49–60% for the non-reasoning model gemma-4-31B.

Table 3. Controls under the grounded detector: negation and trace length

ControlOriginal detectorGrounded
Change in G when sentences stating the absence of a violation are not counted as detections (range, median)−.007 to −.070−.001 to −.040 (−.008)
Share of detections on corrupted traces that rest only on a negated sentence–0.7–7.7% (median 2.8%)
Logistic regression detection ~ corruption + log(length): corruption coefficient–positive in all runs, +1.66 to +3.83 (median +2.71)
Same regression: log(length) coefficient (median)–+0.64
Correlation of firing with log(length) on clean traces, ρup to .56−.06 to +.30 (median +.16)

The grounded detector fires in response to the injected violation, not merely to negated statements or longer traces (Table 3; Appendix D.2). (i) When sentences that state the absence of a violation (e.g., "no duplicates of id 17") are not counted as detections, G changes by a median of −.008. (ii) In a logistic regression of detection on corruption and log trace length, the corruption coefficient is positive in every run (median +2.71), and the median length coefficient is +0.64.

We would also like to clarify that the detector is used only to split silent failures into G and B; SFR and disclosure do not depend on it. SFR, Δα, and disclosure by violation type are computed from abstention and the output alone, so they do not change with the detector. The revision states this scope explicitly in Sec. 4.2.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer SdBs (3/5)

W3. Similarly I'm not sure abstention is always the right response to an integrity violation. SQL normally ignores NULL values during aggregation and the paper notes that CONFLICT-DUP cases can still produce the correct answer. The authors examine silent repairs for duplicated rows but do not report answer correctness consistently across all violation types. I think this distinction matters when interpreting how serious the reported failures actually are.

Thank you for pointing this out. To address this point, we now report answer correctness for every violation type, alongside disclosure.

We would like to clarify that RELIC evaluates whether the violation is disclosed rather than whether the answer is correct. Abstention is one way of disclosing a violation; a model can also disclose it by reporting it in the reporting field (issues). Because the original paper defined SFR as 1 − abstention rate, it could be read as treating abstention as the only correct response. The revision defines SFR by disclosure (Sec. 4.2), and a response that reports the violation and answers is counted as a success. A response that answers without disclosing the violation is counted as a silent failure even if the answer matches the original gold answer, because the user needs to know that a violation was present to decide whether the answer can be used as it is.

Answer correctness complements SFR by showing which answer the user receives when a violation is not disclosed. SFR itself does not depend on correctness. We first define two reference answers. A* is the result of executing the query on the original tables before the corruption is injected, i.e., the original gold answer. à is the result of executing the same query, unmodified, on the tables after the corruption is injected. For example, for NULL-AGG, where NULL values are injected into the aggregated column, à is the aggregate computed with the NULL rows excluded. A response that matches à therefore computes the answer from the corrupted tables as they are, without handling the corruption. For each of the six violation types, the revision reports the fraction of corrupted responses that (i) match A*, (ii) match Ã, and (iii) abstain (Table 4; Appendix E.3).

Table 4. Outcome of corrupted responses by violation type (base condition, gpt-oss-120B, %; unparseable responses are excluded from the denominator; the remainder are other answers)

TypeMatches A*Matches à (result of executing the query on the corrupted tables)Abstention
NULL-AGG0.369.08.7
NULL-FK1.166.36.1
ORPHAN-FK2.560.58.1
SHIFT-PK1.558.311.5
DUP50.96.18.3
CONFLICT-DUP47.50.05.9

For the NULL and referential types, an answer that differs from the original gold answer is returned without a warning; for the duplicate types, 47.5–50.9% of the answers match it, but the violation is not conveyed to the user. For NULL-AGG, NULL-FK, ORPHAN-FK, and SHIFT-PK, 58.3–69.0% of the responses match à and only 0.3–2.5% match A*. For these four types, the corruption removes information needed to compute A* (Sec. 3.3), so A* cannot be obtained from the corrupted tables, and the model returns à without a warning. The result of the SQL aggregation that skips NULL values, mentioned by the reviewer, corresponds to à for NULL-AGG. à is the answer under SQL semantics, but it differs from A*. SQL aggregate functions skip NULL values (SUM(30, NULL) returns 30 [1]), and the result carries no indication that NULL values were present. A user who provides exported tables has not chosen this handling, so without disclosure the user cannot know that the answer received differs from A*. For DUP, the corrupted tables retain all original information, so A* can be obtained, and 50.9% of the responses match A*. For CONFLICT-DUP, two different values remain for the same key, so the tables alone do not determine the answer; the 47.5% of responses that match A* result from the model selecting one of the two values without stating the basis for the selection. The abstention rate is 5.9–11.5% for all six types.

We would therefore suggest that the concern lies less in whether the model abstains than in the fact that it decides how to handle the violation on its own without informing the user. Aggregating with NULL values excluded (Ã) and selecting one of two conflicting values are both ways of handling the violation, and which one is appropriate depends on the intent of the question. If this choice is not disclosed, the user cannot know how the answer was obtained and cannot decide whether to use it as it is. SFR measures this non-disclosure regardless of whether the answer is correct.


[1] Toussaint, Guagliardo, Libkin, and Sequeda. Troubles with Nulls, Views from the Users. VLDB 2022.

(continued in the next comment)

Reviewer_B81hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RELIC, a benchmark for evaluating how LLMs handle integrity violations in multi-table question answering. The authors introduce the Silent Failure Rate (SFR), which measures how frequently models answer questions over corrupted data without disclosing integrity violations. They further distinguish between failures caused by missing violations and those in which the model detects a violation but does not disclose it. Experiments across 20 models suggest that stronger reasoning capabilities do not necessarily lead to better disclosure of data integrity problems.

Critical strengths

S1. The proposed problem, "Sailent Failure", is interesting and new to audience and raise a new question to the community. S2. The proposed benchmark allows the "Silent Failure" to be measured S3. The experiment is thorough and extensive.

Critical weaknesses

W1. Insufficient motivation for tool-free integrity checking. The paper assumes that LLMs must independently detect database integrity violations without external tools. However, most violations considered in RELIC can be reliably detected using standard SQL operations or database validation tools. The authors should better justify the practical relevance of this tool-free setting and consider comparisons with tool-augmented agents.

W2. Evaluation protocol may bias silent failure rates. The prompt and output format may discourage models from detecting or disclosing integrity violations. In particular, the base format lacks a dedicated field for reporting issues, effectively equating non-abstention with silent failure. Although the paper explores prompt and output-format variations, more systematic evaluation is needed to separate genuine model limitations from protocol-induced failures.

W3. Limited realism and generalizability. RELIC focuses on synthetic two-table queries with single-column keys, simple filtering, and numeric aggregation. These settings do not adequately represent complex real-world database workloads. Evaluating more realistic datasets with multiple joins, composite keys, and naturally occurring integrity violations would strengthen the generalizability and practical significance of the findings.

Questions for authors

Q1. How will the "Salient Failure" problem be mitigated, if appropriate tools and prompt is provided to LLM?

7 comments
AuthorsOpenReview ↗

Response to Reviewer B81h (1/4)

We are grateful to the reviewer for the careful reading and helpful suggestions. We respond to each point in turn.

W1. Insufficient motivation for tool-free integrity checking. ... The authors should better justify the practical relevance of this tool-free setting and consider comparisons with tool-augmented agents.

Q1. How will the "Silent Failure" problem be mitigated, if appropriate tools and prompt is provided to LLM?

Thank you for these comments. Since W1 and Q1 concern the same issue, we address them together. As suggested, we compared the models with and without a code-execution tool (Sec. 5.6), and with a prompt that names the violation types to check (Appendix E.6).

We would like to note that database constraint checking is often not available for the inputs that RELIC targets. SQL operations and validation tools reject or report key violations only in a database where key constraints are declared and enforced. Tables given to an LLM are exported CSV or Markdown files, which do not state which columns are keys or which tables they reference [1]. Even in source databases, foreign keys are often undeclared [2][3]: in a survey of open-source projects, 88% of MySQL schemas and 55% of PostgreSQL schemas declare no foreign key [2]. SQLite, used by Spider and BIRD, does not check foreign keys by default [4]. As a result, in the dev splits, 63.6% of BIRD databases and 20.0% of Spider databases contain rows that violate their own declared foreign keys, and 19.8% of BIRD join queries join on a foreign key with such violations (Appendix A).

Even when a violation is detected, how to handle it is the user's decision, so the model should report it to the user. A check query shows which rows violate a constraint, but not what to do with them, because the right handling depends on the question. For example, if a question asks for the total order amount per customer and some orders refer to a customer ID missing from the customer table, the user must decide whether to exclude these orders, list them separately, or not answer. SQL does not ask the user. It applies one fixed handling: an inner join drops unmatched rows and aggregate functions skip NULL values (SUM(30, NULL) returns 30 [5]), and the result shows no sign that rows were dropped. In the setting we evaluate, the user gives the question and the tables to the model and receives only the answer, so only the model can tell the user that a violation exists. RADAR [6] places the same requirement on models for data artifacts within a single table; RELIC extends it to key relations across tables.

New experiment: a code-execution tool increased accuracy but not disclosure. We gave GLM-5 and Kimi-K2 a code-execution tool and used an output format with a reporting field (issues), where the model lists any problems it found along with its answer (Table 1; setup in Appendix E.7).

Table 1. Results with and without a code-execution tool (%, output format with the reporting field; tool-free = no tool, tool-use = with the code-execution tool)

GLM-5Kimi-K2
tool-freetool-usetool-freetool-use
Clean accuracy76.7100.066.796.7
Disclosure (abstention or report of the injected violation type)45.143.3 (McNemar p = .67)58.952.9 (p = .09)
Tool-call rate (corrupted instances)–100.0–78.3
Calls containing an integrity check–18.6–41.4
Check output exposed the violation, model did not report it–15.6–34.1
Answer equals the result of executing the query on the corrupted tables as they are–92.7–88.8

The tool raised clean accuracy by 23.3 and 30.0 points, but disclosure on corrupted inputs (abstention or a report of the injected violation type) did not change significantly (GLM-5 45.1% → 43.3%, Kimi-K2 58.9% → 52.9%). Only 18.6% and 41.4% of tool calls included an integrity check, and 92.7% and 88.8% of answers matched the result of running the query directly on the corrupted tables. Table 2 breaks the results down by violation type.

Table 2. Disclosure by violation type (%, all three conditions use the output format with the reporting field; disclosure = abstention or a report of the injected violation type; tool-free = no tool, tool-use = with the code-execution tool, explicit = instruction enumerating the six violation types)

TypeGLM-5Kimi-K2
tool-freetool-usetool-free + explicittool-freetool-usetool-free + explicit
DUP93.2100.0100.097.8100.0100.0
ORPHAN-FK19.211.588.136.423.698.3
SHIFT-PK15.213.081.041.519.593.0

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer B81h (2/4)

Q1: a reporting field alone resolves silent failure only for duplicated keys; for referential-integrity violations, only an instruction that names the violation types helps, and the tool does not (Table 2). A duplicated key (DUP) can be checked within one table and is disclosed in 93–98% of cases with the reporting field alone. Orphaned and shifted keys (ORPHAN-FK, SHIFT-PK) can be checked only by comparing keys across the two tables and are disclosed in only 15–42% of cases with the reporting field alone. The tool does not raise this rate (12–24%), whereas an instruction listing the six violation types raises it to 81–98%. In other words, the two models compare keys across tables only when the prompt names the violation to look for.


[1] Narayan et al. Can Foundation Models Wrangle Your Data? VLDB 2022.

[2] Coelho, Aillos, Pilot, and Valeev. On the Quality of Relational Database Schemas in Open-source Software. International Journal on Advances in Software 4(3&4), 2011.

[3] Zhang, Hadjieleftheriou, Ooi, Procopiuc, and Srivastava. On Multi-Column Foreign Key Discovery. PVLDB 3(1), 2010.

[4] SQLite. Foreign Key Support. SQLite documentation.

[5] Toussaint, Guagliardo, Libkin, and Sequeda. Troubles with Nulls, Views from the Users. VLDB 2022.

[6] Gu et al. RADAR: Benchmarking Language Models on Imperfect Tabular Data. NeurIPS 2025.

W2. Evaluation protocol may bias silent failure rates. The prompt and output format may discourage models from detecting or disclosing integrity violations. In particular, the base format lacks a dedicated field for reporting issues, effectively equating non-abstention with silent failure. Although the paper explores prompt and output-format variations, more systematic evaluation is needed to separate genuine model limitations from protocol-induced failures.

Thank you for this comment. Following this suggestion, we evaluated the models under six combinations of instruction level and output format.

We would like to clarify that a non-abstaining response under the base format is a silent failure by definition: it answers a corrupted input without disclosing the violation. Under the base format, such a response gives the user no sign that the input is corrupted. The base format is the standard format of table QA and text-to-SQL benchmarks, in which a question and tables are given and a single answer is returned [1][2], with two additions: a warning that the tables may contain integrity violations, and an explicit permission to abstain, which is the only means of disclosure in this format (Sec. 4.1, Appendix C.2). Since neither the standard format nor real-use prompts include these additions, the base condition is more favourable to the model than both. Even so, for every model the abstention rates on corrupted and clean inputs differ by at most 4.3 points (Table 1 of the paper), which indicates that the models rarely use abstention to disclose a violation even though it is explicitly permitted.

Table 3. The six evaluation conditions and where the results of each are reported

Instruction level \ Output formatanswer-onlyanswer+issues
baseTable 1 of the paper; upper bound of SFRTable 4
informedTable 2 of the paperTable 4
explicitTable 2 of the paperTable 4; lower bound of SFR

We agree that a single condition cannot separate model limitations from protocol-induced failures. The six conditions cross three instruction levels with two output formats (Table 3; Appendix E.6). Base warns of possible violations and permits abstention but does not ask for a check; informed adds a request to verify referential integrity; explicit additionally lists all six violation types (Sec. 5.4, Appendix C.1). The output is either answer-only or answer+issues, which adds a reporting field (issues) for the problems the model found. In every condition, a silent failure is a response that answers without abstaining and, where a reporting field exists, without reporting the violation there. Base × answer-only gives the model the least help in checking for and reporting a violation, so its SFR is the upper bound; explicit × answer+issues gives the most help, so its SFR is the lower bound.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer B81h (3/4)

SFR remains at 41–80% with the reporting field alone and at 7–66% at the lower bound, depending on the model. We extended the answer+issues format from three models in the original paper to eleven, at all three instruction levels (Table 4; Sec. 5.5). The reporting field alone reduces SFR by 6–59 points in all eleven models, and SFR generally decreases as the instruction becomes more specific. At the lower bound, SFR is 10.9% for GLM-5, 6.8% for Kimi-K2, and 26–66% for the other nine models. Since LLM outputs are sensitive to prompt format [3], the revision reports SFR for every combination of instruction level and output format rather than under a single condition (Sec. 4.1).

Table 4. SFR per condition (%, corrupted inputs; base × answer-only from Table 1 of the paper)

Modelbase × answer-onlybase × answer+issuesinformed × answer+issuesexplicit × answer+issues
GLM-599.254.923.710.9
Kimi-K299.741.125.36.8
gpt-oss-120B93.252.348.832.7
gpt-oss-20B92.372.465.361.1
Qwen3-32B93.958.554.043.0
Qwen3-8B92.476.363.455.1
gemma-4-31B96.866.857.926.4
R1-Distill-7B95.364.162.966.2
QwQ-32B92.464.051.149.9
R1-Distill-70B86.079.667.763.2
Llama-3.3-70B88.765.844.332.0

At the lower bound, SFR on DUP falls to 0–16% in seven of eleven models, whereas SFR on ORPHAN-FK and SHIFT-PK falls to 2–19% only for GLM-5 and Kimi-K2 and remains at 46–75% for the other nine (Table 5). DUP can be verified within a single table, whereas ORPHAN-FK and SHIFT-PK require comparing keys across the two tables.

Table 5. SFR by violation type at the lower bound (explicit × answer+issues, %)

ModelNULL-AGGNULL-FKORPHAN-FKSHIFT-PKDUPCONFLICT-DUP
GLM-510.522.411.919.00.00.0
Kimi-K210.521.11.77.00.00.0
gpt-oss-120B33.153.646.246.82.114.6
gpt-oss-20B69.285.974.474.715.833.7
Qwen3-32B52.649.146.150.08.547.2
Qwen3-8B51.538.258.957.343.078.7
gemma-4-31B6.226.760.863.60.01.2
R1-Distill-7B60.955.471.266.369.573.8
QwQ-32B46.865.551.454.123.755.3
R1-Distill-70B64.473.171.872.140.856.9
Llama-3.3-70B24.626.751.748.14.636.2

The six conditions separate the part of silent failure that the prompt and output format can reduce from the part they cannot, and show which intervention each violation type requires. The lower-bound prompt is not proposed for practical use, since only a user who already knows which violation types to check could write it; it measures how much silent failure remains under the most favourable prompt and output format. (i) A reporting field reduces SFR in all eleven models, and seven models then report 72–99% of duplicated keys. (ii) Naming the violation types in the instruction reduces SFR further: for GLM-5 and Kimi-K2, SFR is 24–25% when asked to verify referential integrity and 7–11% when the six types are listed. (iii) For the other nine models, 46–75% of ORPHAN-FK and SHIFT-PK violations remain undisclosed even when the types are named; this part is a model limitation rather than protocol-induced, and it requires improving the model's detection ability. Applying the same six conditions to a new model or prompt shows, on the same basis, which of these cases it falls into.


[1] Yu et al. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. EMNLP 2018.

[2] Li et al. Can LLM Already Serve as a Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. NeurIPS 2023.

[3] Sclar et al. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design. ICLR 2024.

(continued in the next comment)

Comments on the paper

1 comment
AuthorsOpenReview ↗

Kind Reminder Before the Discussion Deadline

Dear Reviewers,

We sincerely thank you for your time and effort in reviewing our paper.

To make our responses easier to review, we posted a brief summary at the end of each review thread, which shows in a single table how each of your comments was addressed. If your schedule is tight, we would greatly appreciate it if you could take a quick look at that summary.

We hope our responses have addressed your concerns, and we would be happy to clarify anything further. Thank you again for your thoughtful and constructive feedback.

Best regards,

Authors of Submission 39096

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.