- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper formulates budgeted error detection for table question answering (TQA) and introduces an Online, query-aware detection policy that interleaves detection with query execution. It develops a data error taxonomy and proposes BED-TQA, a configurable benchmark derived from four TQA datasets. Experiments show that Online detection achieves up to 33.2% higher answer accuracy than Offline detection under limited budgets, while Offline becomes more cost-effective when repairs can be reused across multiple queries.
Critical strengths
- The paper is well-written and well-organized, with clear motivation, problem formulation, and explanations of the proposed approach.
- The paper provides extensive experiments across multiple datasets and settings, demonstrating substantial improvements over Offline detection.
- The proposed query-aware Online detection approach effectively prioritizes query-relevant errors and improves TQA accuracy under limited detection budgets.
Critical weaknesses
Limited realism of synthetic errors. Although BED-TQA validates injected errors, it does not sufficiently account for database type and integrity constraints (e.g., NOT NULL, CHECK, and FOREIGN KEY). Some corruptions may be rejected by stricter database schemas, raising concerns about their realism and generalizability. 2. Lack of cross-query repair reuse in Online. Online performs detection and repair independently for each query without persisting results. Repeated queries over the same data may incur redundant LLM costs. A comparison with a stateful Online approach that caches validated repairs would provide a fairer long-term cost evaluation. 3. Limited large-scale database evaluation. Experiments mainly use short-table subsets containing 1,572 instances from 249 underlying sources. Scalability is primarily tested by adding irrelevant columns, leaving performance on genuinely large databases, dynamic updates, and sustained query workloads unclear. 4. Inconsistent token reduction claims. The claimed 59.2% reduction in LLM token usage conflicts with the reported averages in Section 6.4 (5,773 vs. 3,324 tokens), which imply a 42.4% reduction. The authors should clarify the discrepancy and ensure consistent reporting.
Questions for authors
- How do the injected errors reflect real-world database constraints and error distributions?
- Have the authors considered a stateful Online approach that caches and reuses repairs across queries?
- How would the proposed method perform on large-scale databases with millions of rows, multiple tables, and dynamic updates?
- How is the claimed 59.2% reduction calculated, given that the reported token usage and monetary costs imply reductions of 42.4% and 31.7%, respectively?
4 comments
W3/Q3
Thank you for highlighting these settings. We added three controlled experiments:
Table size. Our benchmark contains few naturally large tables, so to increase the size of the database we increased table sizes by replicating randomly selected rows and measured the resulting detection-token usage:
| Rows per table | Offline detection tokens | Online detection tokens |
|---|---|---|
| 100 | 7,727 | 2,040 |
| 1,000 | 39,391 | 10,865 |
| 10,000 | 394,699 | 108,618 |
Both policies become more expensive as tables grow, but Online uses approximately 72%–74% fewer detection tokens on average.
Number of tables. Similar as before, our benchmark also does not include many database instances with a several number of tables. Therefore we used a sample of BIRD question-seed instances and added unrelated tables while holding the query-relevant data, resolved query scope and original detection budget fixed.
| Total tables | Offline accuracy | Online accuracy | Offline detection tokens | Online detection tokens |
|---|---|---|---|---|
| 4 | 44.7% | 74.9% | 17,462 | 5,028 |
| 8 | 42.0% | 74.9% | 20,470 | 5,028 |
| 16 | 41.2% | 74.9% | 21,539 | 5,028 |
| 32 | 40.8% | 74.9% | 21,588 | 5,028 |
As unrelated tables accumulate, Offline spends more actions outside the query scope: accuracy falls from 44.7% to 40.8% while token use increases. Online remains unchanged because its scope and detector inputs are fixed.
Dynamic updates. We also compared incremental Offline and stateful Online (proposed by the reviewer and proved to be effective) under different update frequencies. Both retain previous repairs. After an update, Offline reruns detection on the changed tables. Online performs the needed checks within each incoming question’s plan-derived scope and reuses cached results only when the table contents, detector settings, and any required question context still match. Each workload starts with half of its corruption operations applied. We then introduce additional corruption batches after every question or every two questions, or leave the initial data unchanged.
| New-corruption schedule | Incremental Offline accuracy | Stateful Online accuracy | Incremental Offline tokens/question | Stateful Online tokens/question |
|---|---|---|---|---|
| No updates | 76.0% | 91.1% | 1,118 | 3,504 |
| After every two questions | 75.2% | 89.5% | 6,268 | 3,781 |
| After every question | 74.6% | 87.4% | 12,502 | 3,963 |
With new errors after every question, Online maintains higher accuracy (87.4% versus 74.6%) at lower token cost (3,963 versus 12,502). Repeated maintenance erodes Offline's reuse advantage. More frequent arrivals also add more errors in these workloads, so the results reflect both effects. Once again we thank the reviewers for suggesting more real-world settings for evaluation. These experiments further strengthen our initial claims and we plan to include them in the paper.
W4/Q4
We thank the reviewer for their careful attention to these numbers. The reported averages imply 42.4% fewer LLM tokens. The 59.2% figure was intended to describe the maximum dataset-level reduction observed on BIRD. We will correct the claim throughout the paper to: “Online uses 42.4% fewer LLM tokens on average than Offline, with a maximum dataset-level reduction of 59.2% on BIRD.” We will also distinguish token reductions from monetary-cost reductions.
Thanks for your detailed feedback.
Thank you for taking the time to consider our response! We would be happy to address any remaining questions or concerns. If our clarifications have addressed your main concerns, we would appreciate your reconsideration of the score.
We thank the reviewer for their constructive feedback and careful attention to the results. Please find the our response to your feedback below. We will incorporate these clarifications and results into the revised manuscript:
W1/Q1
We examined Hospital, Flights, Beers and Rayyan, used in Raha (Mahdavi et al., 2019) and Baran (Mahdavi & Abedjan, 2020). Their reported cell-level error rates span 3%–30%: Hospital 3%, Rayyan 9%, Beers 16% and Flights 30% (Mahdavi & Abedjan, 2020, Table 2). The labeled DGov-X variants used in MaTElDa have injected rates of 2%–16% (Ahmadi et al., 2025, Table 1). Error rates and composition vary substantially across domains, tasks and injection procedures. This motivates BED-TQA's configurable corruption rates, error types and mixtures.
Our default setting corrupts approximately 10% of cells, distributed roughly uniformly across applicable error types. This balance prevents dominant error types from skewing aggregate results. We do not assume that real-world errors follow a uniform distribution, and our sensitivity experiments vary error density and composition.
Database constraints provide useful safeguards, but declared constraints can miss both syntactic and semantic errors. NOT NULL rejects SQL NULL but permits missing-value markers such as -99. Typos and misfielded values can still satisfy a column's type and declared checks. CHECK enforces ranges, vocabularies or formats only when the relevant rules are known and encoded. Synonyms and abbreviations can also satisfy these checks while failing to match a query's intended value.
These limitations have motivated decades of data-cleaning research. Our seven error types build on Bhadauria et al. (2026), as described in Section 3.2. In spreadsheets (Chen et al., 2025), data lakes and web-table collections, we may have only the table contents, without declared database constraints.
We agree that database constraints can be useful. We therefore audited the original source schemas in the datasets and only found primary keys, foreign keys and NOT NULL constraints. These constraints identify key inconsistencies and explicit NULL values. However, none of the audited schemas provides CHECK constraints specifying allowed formats, vocabularies or value ranges. The available constraints therefore support some error detection, but do not encode the domain-specific rules needed to identify many of the errors considered in our benchmark.
W2/Q2
Thank you for suggesting this important extension. Stateful Online persists repaired cell values and caches detection results across questions on the same source.
We implement this caching mechanism as follows. A cached result is specific to the source, table, column and error type, and is reused only when the complete table contents and detector settings match. A typo result cannot replace a synonym result. Context-sensitive detectors for synonyms, abbreviations and embedded values also require matching question, plan and literal context. Syntactic detectors can reuse results across questions because they do not use question context in this configuration.
We evaluate this caching mechanism using the paper’s reuse workload (Section 6.4):
(A) denotes the number of questions answered using the same underlying database.
Stateful Online improves accuracy and reduces token usage on all three datasets, although SQuALL's token savings are small. Reuse therefore benefits Online while preserving an accuracy–cost trade-off. We will make sure to include the insights from this experiment to the paper in the revision.