acceptodds
Under review as a conference paper at ICLR 2027

Budgeted Data Error Detection for Table Question Answering

Abstract

Table question answering (TQA) can fail because of errors in the underlying data, yet data errors remains a largely overlooked source of TQA failures. Error detection is typically performed Offline, targeting all data in the database. However, when detection relies on costly LLMs, limited budgets make it important to allocate detection effort effectively. Queries provide signals for focusing this effort on data that can affect the answer, motivating Online, query-aware detection. We formulate budgeted error detection for TQA and introduce an Online detection policy that leverages the query to interleave detection with execution. We develop a data error taxonomy and propose BED-TQA, a configurable benchmark for budgeted error detection derived from four TQA datasets. We evaluate detection policies across budgets, error distribution, query scope, database size and workload conditions. Given the same budget to scan all columns once, TQA with Online detection achieves up to 33.2% higher answer accuracy than Offline detection, while using 59.2% fewer LLM tokens. Online detection remains competitive across most settings, particularly under tight budgets. Offline becomes more cost-effective only when repairs can be reused across multiple queries. Overall, our results challenge the popular assumption that data errors should be detected and repaired Offline: Online detection can allocate budget more effectively toward relevant data.

Primary area:
datasets and benchmarks
Keywords:
Table Question Answering, Data Cleaning, Data Quality, Query-Aware Cleaning, Error Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YYVXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates budgeted error detection for table question answering (TQA) and introduces an Online, query-aware detection policy that interleaves detection with query execution. It develops a data error taxonomy and proposes BED-TQA, a configurable benchmark derived from four TQA datasets. Experiments show that Online detection achieves up to 33.2% higher answer accuracy than Offline detection under limited budgets, while Offline becomes more cost-effective when repairs can be reused across multiple queries.

Critical strengths
  1. The paper is well-written and well-organized, with clear motivation, problem formulation, and explanations of the proposed approach.
  2. The paper provides extensive experiments across multiple datasets and settings, demonstrating substantial improvements over Offline detection.
  3. The proposed query-aware Online detection approach effectively prioritizes query-relevant errors and improves TQA accuracy under limited detection budgets.
Critical weaknesses

Limited realism of synthetic errors. Although BED-TQA validates injected errors, it does not sufficiently account for database type and integrity constraints (e.g., NOT NULL, CHECK, and FOREIGN KEY). Some corruptions may be rejected by stricter database schemas, raising concerns about their realism and generalizability. 2. Lack of cross-query repair reuse in Online. Online performs detection and repair independently for each query without persisting results. Repeated queries over the same data may incur redundant LLM costs. A comparison with a stateful Online approach that caches validated repairs would provide a fairer long-term cost evaluation. 3. Limited large-scale database evaluation. Experiments mainly use short-table subsets containing 1,572 instances from 249 underlying sources. Scalability is primarily tested by adding irrelevant columns, leaving performance on genuinely large databases, dynamic updates, and sustained query workloads unclear. 4. Inconsistent token reduction claims. The claimed 59.2% reduction in LLM token usage conflicts with the reported averages in Section 6.4 (5,773 vs. 3,324 tokens), which imply a 42.4% reduction. The authors should clarify the discrepancy and ensure consistent reporting.

Questions for authors
  1. How do the injected errors reflect real-world database constraints and error distributions?
  2. Have the authors considered a stateful Online approach that caches and reuses repairs across queries?
  3. How would the proposed method perform on large-scale databases with millions of rows, multiple tables, and dynamic updates?
  4. How is the claimed 59.2% reduction calculated, given that the reported token usage and monetary costs imply reductions of 42.4% and 31.7%, respectively?
4 comments

We thank the reviewer for their constructive feedback and careful attention to the results. Please find the our response to your feedback below. We will incorporate these clarifications and results into the revised manuscript:

W1/Q1

We examined Hospital, Flights, Beers and Rayyan, used in Raha (Mahdavi et al., 2019) and Baran (Mahdavi & Abedjan, 2020). Their reported cell-level error rates span 3%–30%: Hospital 3%, Rayyan 9%, Beers 16% and Flights 30% (Mahdavi & Abedjan, 2020, Table 2). The labeled DGov-X variants used in MaTElDa have injected rates of 2%–16% (Ahmadi et al., 2025, Table 1). Error rates and composition vary substantially across domains, tasks and injection procedures. This motivates BED-TQA's configurable corruption rates, error types and mixtures.

Our default setting corrupts approximately 10% of cells, distributed roughly uniformly across applicable error types. This balance prevents dominant error types from skewing aggregate results. We do not assume that real-world errors follow a uniform distribution, and our sensitivity experiments vary error density and composition.

Database constraints provide useful safeguards, but declared constraints can miss both syntactic and semantic errors. NOT NULL rejects SQL NULL but permits missing-value markers such as -99. Typos and misfielded values can still satisfy a column's type and declared checks. CHECK enforces ranges, vocabularies or formats only when the relevant rules are known and encoded. Synonyms and abbreviations can also satisfy these checks while failing to match a query's intended value.

These limitations have motivated decades of data-cleaning research. Our seven error types build on Bhadauria et al. (2026), as described in Section 3.2. In spreadsheets (Chen et al., 2025), data lakes and web-table collections, we may have only the table contents, without declared database constraints.

We agree that database constraints can be useful. We therefore audited the original source schemas in the datasets and only found primary keys, foreign keys and NOT NULL constraints. These constraints identify key inconsistencies and explicit NULL values. However, none of the audited schemas provides CHECK constraints specifying allowed formats, vocabularies or value ranges. The available constraints therefore support some error detection, but do not encode the domain-specific rules needed to identify many of the errors considered in our benchmark.

W2/Q2

Thank you for suggesting this important extension. Stateful Online persists repaired cell values and caches detection results across questions on the same source.

We implement this caching mechanism as follows. A cached result is specific to the source, table, column and error type, and is reused only when the complete table contents and detector settings match. A typo result cannot replace a synonym result. Context-sensitive detectors for synonyms, abbreviations and embedded values also require matching question, plan and literal context. Syntactic detectors can reuse results across questions because they do not use question context in this configuration. 

We evaluate this caching mechanism using the paper’s reuse workload (Section 6.4):

DatasetAOnline accuracyStateful Online accuracyOnline tokens/questionStateful Online tokens/question
BIRD1482.5%86.2%8,6017,512
SQuALL485.4%88.2%1,9311,913
WikiSQL481.7%83.9%1,7281,539

(A) denotes the number of questions answered using the same underlying database.

Stateful Online improves accuracy and reduces token usage on all three datasets, although SQuALL's token savings are small. Reuse therefore benefits Online while preserving an accuracy–cost trade-off. We will make sure to include the insights from this experiment to the paper in the revision.

W3/Q3

Thank you for highlighting these settings. We added three controlled experiments:

Table size. Our benchmark contains few naturally large tables, so to increase the size of the database we increased table sizes by replicating randomly selected rows and measured the resulting detection-token usage:

Rows per tableOffline detection tokensOnline detection tokens
1007,7272,040
1,00039,39110,865
10,000394,699108,618

Both policies become more expensive as tables grow, but Online uses approximately 72%–74% fewer detection tokens on average. 

Number of tables. Similar as before, our benchmark also does not include many database instances with a several number of tables. Therefore we used a sample of BIRD question-seed instances and added unrelated tables while holding the query-relevant data, resolved query scope and original detection budget fixed. 

Total tablesOffline accuracyOnline accuracyOffline detection tokensOnline detection tokens
444.7%74.9%17,4625,028
842.0%74.9%20,4705,028
1641.2%74.9%21,5395,028
3240.8%74.9%21,5885,028

As unrelated tables accumulate, Offline spends more actions outside the query scope: accuracy falls from 44.7% to 40.8% while token use increases. Online remains unchanged because its scope and detector inputs are fixed. 

Dynamic updates. We also compared incremental Offline and stateful Online (proposed by the reviewer and proved to be effective) under different update frequencies. Both retain previous repairs. After an update, Offline reruns detection on the changed tables. Online performs the needed checks within each incoming question’s plan-derived scope and reuses cached results only when the table contents, detector settings, and any required question context still match. Each workload starts with half of its corruption operations applied. We then introduce additional corruption batches after every question or every two questions, or leave the initial data unchanged.

New-corruption scheduleIncremental Offline accuracyStateful Online accuracyIncremental Offline tokens/questionStateful Online tokens/question
No updates76.0%91.1%1,1183,504
After every two questions75.2%89.5%6,2683,781
After every question74.6%87.4%12,5023,963

With new errors after every question, Online maintains higher accuracy (87.4% versus 74.6%) at lower token cost (3,963 versus 12,502). Repeated maintenance erodes Offline's reuse advantage. More frequent arrivals also add more errors in these workloads, so the results reflect both effects. Once again we thank the reviewers for suggesting more real-world settings for evaluation. These experiments further strengthen our initial claims and we plan to include them in the paper.

W4/Q4

We thank the reviewer for their careful attention to these numbers. The reported averages imply 42.4% fewer LLM tokens. The 59.2% figure was intended to describe the maximum dataset-level reduction observed on BIRD. We will correct the claim throughout the paper to: “Online uses 42.4% fewer LLM tokens on average than Offline, with a maximum dataset-level reduction of 59.2% on BIRD.” We will also distinguish token reductions from monetary-cost reductions.

Thanks for your detailed feedback.

Thank you for taking the time to consider our response! We would be happy to address any remaining questions or concerns. If our clarifications have addressed your main concerns, we would appreciate your reconsideration of the score.

Reviewer_QxCqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses online data error detection under a limited budget of detection actions. The paper points out that most existing table question answering (TQA) studies assume clean tables and introduces BED-TQA, a benchmark constructed by injecting errors into existing TQA datasets.

The benchmark covers three main error categories: Syntactic, Semantic, and Embedded Value. Syntactic errors include missing values, typographical errors, format inconsistencies, and misfielded values. Semantic errors include synonyms, such as "finalized" versus "completed," and abbreviations. Embedded-value errors occur when a value is present but embedded in surrounding text. These types of errors can occur in real-world data, making this an important problem.

LLMs can detect these errors, but their high inference cost makes broad error detection expensive. To address this issue, the authors propose an online error detection policy under a limited action budget. Each detection action applies a detector to a selected column, combining rule-based deterministic screening with an LLM-based check. The online policy uses the query plan and intermediate execution results to select relevant actions. The authors compare this approach with offline detection under the same budget constraints to evaluate how effectively each policy allocates its budget to relevant data.

Critical strengths
  • This paper addresses an important problem that can occur in real-world TQA systems. While previous TQA robustness studies have mainly focused on perturbations to schemas, this paper also considers errors in the underlying table values.
  • The proposed benchmark covers reasonable error types that can occur in real-world data.
Critical weaknesses
  • The online error detection method has limited technical novelty.
  • It seems that this problem could be solved using rule-based methods without requiring LLMs. If so, it may be sufficient to address the problem using only offline detection rather than online detection. The proposed method uses LLMs to parse column names and check the results of deterministic screening. However, I wonder whether all of these tasks could be performed using rule-based methods. For instance, an Abstract Syntax Tree (AST) could be used to parse column names. Moreover, the paper does not provide results without LLM-based checking.
Questions for authors
  • In the paper, the budget is defined based on the number of detection actions. Do all detection actions have the same computational cost? If not, could the budget be defined by considering these cost differences?

  • What would the detection performance be if the authors used only rule-based deterministic screening without LLM-based checking?

  • If the generated SQL is incorrect, the proposed method does not appear to use its detection budget to correct the SQL itself. Although Appendix F evaluates generated SQL queries, the answer accuracy remains substantially lower than when using reference SQL. How does the proposed method handle SQL generation errors that may cause the detection budget to be allocated to irrelevant data?

4 comments

We thank the reviewer for their constructive feedback. Please find the our response to your feedback below. We will incorporate these clarifications and results into the revised manuscript:

W1

Our contribution centers on budgeted, query-aware error detection for TQA, BED-TQA and the systematic analysis of detection policies. The formulation makes the allocation of detection actions across columns and error types explicit, with downstream answer accuracy as the objective.

We challenge the assumption that error detection should always be performed offline, without claiming a universally superior detection method. Using a shared bank of existing detectors, we compare Offline with Online, Question-only and Hybrid policies that incorporate detection at query time. We examine how query context, detection budgets and workload properties affect detection quality, answer accuracy and token cost, identifying conditions under which these alternatives improve accuracy and reduce cost.

Our benchmark framework, BED-TQA introduces controlled errors into existing TQA datasets and provides reference queries, clean values and answer-supporting scopes. Its configurable error types and rates allow practitioners to construct benchmarks and evaluate TQA pipelines under data errors.

W2/Q2

Our pipeline already uses deterministic AST parsing to extract referenced columns, operator roles and filter literals (Section 5, page 5, line 259). LLM checks assess candidate data values. For example, parsing identifies a filter on “Human Resources”, but cannot establish that a stored value “HR” has the same meaning.

To address their concern around performance with rule-based screening, we replaced the full detector bank with a rules-only bank. The rules cover missing-value markers, spelling similarity, format patterns, misfielded values, embedded values and unambiguous abbreviations. Our rules-only bank omits synonym detection because no dictionary of equivalent terms is supplied to this baseline.

PolicyRules-only answer accuracyFull-bank answer accuracy
Offline47.5%51.4%
Question-only55.7%68.5%
Online57.2%68.6%
Hybrid49.1%58.6%

Online outperforms Offline even with rules-only detection, showing that query-aware allocation remains useful without LLM-based detector calls. The full detector bank improves answer accuracy for every policy. We will include this rules-only comparison in the revised manuscript.

Q1

We agree that equal action budgets do not imply equal token costs. A detection action applies one detector to one column, with a cost that depends on the column contents, detector type and supplied context. 

To address the reviewer’s concern, we also evaluated the policies under a token budget rather than a budget limit on the number of detector–column pairs tested. We added an explicit token cap which covers question linking and detection. Before each LLM call, the runner reserves a conservative input-token bound plus the maximum output allowance and permits the call only if it fits within the remaining budget.

Token cap per questionOffline accuracyQuestion-only accuracyOnline accuracyHybrid accuracy
4,00047.6%46.4%46.4%46.4%
8,00059.4%66.9%69.9%51.3%
16,00066.2%72.8%81.2%64.2%

Online has the highest accuracy at the 8,000- and 16,000-token caps. At 4,000 tokens, the conservative reservation rule blocks question linking in 97% of Online instances, leaving little opportunity for detection. Online’s advantage therefore appears at the larger tested budgets, where linking and subsequent detection can proceed.

Q3

Our method detects data errors but does not correct SQL-generation errors. Correcting SQL-generation errors is orthogonal to our focus on budgeted data-error detection. Given a query plan, our approach allocates a limited detection budget for data errors that can affect the query answer. An incorrect plan can nevertheless direct detection toward irrelevant columns or produce an incorrect answer even after data repair. We therefore use reference plans in the main experiments to isolate detection quality from planning errors (Also see our response to R1-W1/Q1 for further discussion).

Appendix F evaluates generated plans and shows that planning errors substantially reduce answer accuracy. Our extended comparison uses the same generated SQL across all policies (also see R1 W1/Q2). Online achieves 42.6% accuracy, compared with 33.3% for Offline and 44.4% for Question-only. Online therefore retains an advantage over Offline, but planning errors remain a limitation of the overall pipeline. R1 W1/Q1 also reports the comparison across planning LLMs.

Reviewer_QxCqOpenReview ↗

Official Comment by Reviewer QxCq

Thank you for the detailed response, which addresses some of my previous concerns.

I believe BED-TQA addresses an important problem that can arise in real-world TQA systems. As discussed in your response to Reviewer YYVX (W3/Q3), although the database sizes and experimental settings may not fully reflect real-world environments, I believe the benchmark captures real-world TQA scenarios reasonably well, as it builds upon existing public TQA datasets and covers data error types commonly encountered in practice. I therefore consider it a meaningful contribution.

Given these considerations, I am considering raising my rating to Weak Accept. However, I still have some concerns about whether the problem is sufficiently challenging beyond its practical importance. In particular, I would like to better understand whether the baselines are reasonably constructed and how much room remains for improving detection policies under the proposed budget constraints.

I would appreciate clarification on the following two points.

1. Detection actions and computational cost of the Hybrid policy

The additional rules-only experiments in your response to W2/Q2 partially address my previous concerns. However, I would appreciate further clarification regarding the Hybrid policy.

According to Appendix B, Hybrid allocates its budget between Offline detection for syntactic errors and Online detection for semantic errors. As I understand it, the same column may be examined by both branches.

In this case, could you clarify whether columns already examined by the Offline branch are processed again by the Online branch, and whether information obtained from earlier detection actions is reused? While applying different detectors to the same column may be necessary, it is unclear to me how much redundant processing occurs between the two branches.

In particular, the generated-SQL experiment reported in your response to Reviewer 9urr (W1/Q2) shows that Hybrid consumes more tokens than Offline (6,939 vs. 6,455). Could you clarify whether this additional cost comes from overlapping column processing, additional question/schema linking, or differences in detector costs? If possible, reporting the number of detection actions actually performed and the token consumption of each branch would help clarify this behavior.

2. Detection budget and potential for further improvement

The additional token-budget experiments in your response to Q1 are helpful, as they consider the actual computational cost rather than simply counting detection actions. However, I would also be interested in the number of detection actions actually performed under each token budget.

Appendix D.2 already shows that answer accuracy improves as the detection budget increases. Nevertheless, these results do not fully clarify how reasonable the default budget B=CB=C is, or how much additional improvement could be achieved through better action selection under the same budget.

For example, since multiple error types may occur within a single column, several detection actions may be required to identify the errors relevant to a query. This raises the question of whether some instances are inherently difficult to solve under the default budget, regardless of how effectively the actions are selected.

I understand that obtaining an optimal detection policy may be difficult. However, would it be possible to provide an oracle action-selection baseline or an approximate upper bound using ground-truth error information, while keeping the detector bank and detection budget fixed?

Such an analysis would help distinguish whether the current performance is primarily limited by the available detection budget or by the efficiency of the action-selection policy.

Thank you for your thoughtful follow-up, for recognizing the benchmark’s contribution, and for considering a higher rating. Given the limited time before today’s deadline, we have made every effort to address your remaining concerns. We provide our responses below:

1. Hybrid processing and cost

In the generated-plan setting, Hybrid runs Offline detection and repair first. Its Online branch then checks the repaired data for semantic errors in an interleaved manner. The branches examine 0.64 shared columns per instance, but no detector–column pair is repeated, as they check different error types.

Hybrid componentMean detection actionsMean tokens
Offline syntactic detection8.225,181
Online semantic detection2.44218
Schema linking—815
SQL planning—682
Total10.666,896

This rerun uses 6,896 tokens, compared with the previously reported 6,939. Hybrid uses fewer detection tokens (first two rows of the table above) than Offline (5,399 vs. 5,773), and both incur the same planning cost. However, compared to Offline, Hybrid’s additional 815 schema-linking tokens make its total cost higher (6,896 vs. 6,455).

2. Completed actions and oracle action selection

Completed actions under token cap:

Using the same question-seed instances and reference plans from the previously requested token-budget experiment, we report mean completed detection actions under each token cap:

Token capOfflineQuestion-onlyOnlineHybrid
4,0001.960.000.000.07
8,00017.5610.8910.657.91
16,00029.9315.5013.7220.13

At the 4,000-token cap, the conservative budget check blocked the required LLM calls, so Question-only and Online completed no detection actions.

Oracle action selection:

The oracle uses the ground-truth answer to select detector–column actions that maximize answer accuracy within the budget. We compare B=C with no action cap:

Action selectionAction budgetAnswer accuracy
OnlineB=C68.6%
Oracle selectionB=C81.1%
Oracle selectionNo action cap81.1%

Oracle action selection improves answer accuracy by 12.4 percentage points at B=C. Removing the action cap recovers no additional correct answers, so every instance solved by the uncapped oracle is already solvable within B=C. Current policies can still benefit from larger budgets though because they may spend actions on checks that do not help answer the question. 

Thanks again for your careful evaluation of our paper. We will make sure to include these clarifications in our revision.

Reviewer_9urrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies budgeted data-error detection for table question answering (TQA), where a limited number of detector-column actions are allocated to identify and repair corrupted cells that may affect a query answer. It introduces BED-TQA, a configurable benchmark built by injecting seven error types into instances from MMQA, WikiSQL, SQuALL, and BIRD. The paper compares a query-agnostic Offline policy with Question-only and interleaved query-aware Online policies. It evaluates answer accuracy, error-detection quality, and LLM-token cost under different detection budgets, query contexts, corruption densities, database sizes, query scopes, and workload-reuse levels. It also reports experiments with generated query plans, an alternative detector model, and a holistic repair model.

Critical strengths
  1. Clear problem formulation. The paper makes the detection budget explicit and separates detection from query planning and repair. The Offline, Question-only, and Online comparison provides a useful framework for studying query-conditioned cleaning.

  2. Useful benchmark instrumentation. BED-TQA records clean and corrupted values, error types, answer-supporting cells, and restoration information. The scope-validation tests cover selections, joins, aggregation, anti-joins, ordering, and nested queries.

  3. Broad controlled analysis. The paper evaluates budgets, context, corruption density, distractor columns, query scope, workload reuse, generated plans, detector models, and holistic repair, giving a detailed view of cost-quality trade-offs.

Critical weaknesses
  1. The headline comparison is conditional on oracle query information. The primary evaluation supplies an executable reference plan and query literals to the query-aware policy. This is a valid way to isolate data-error detection, but it means the reported 33.2% improvement over Offline does not measure an end-to-end TQA system that must first obtain a correct plan. This distinction matters because the generated-plan experiment reports only 61.7% plan accuracy and 42.6% Online answer accuracy after repair, versus 68.6% with reference plans (Table 24). The generated-plan results also omit Offline, so it remains unclear whether the policy ordering changes once all methods use the same imperfect plan.

  2. The advantage of interleaving is not fully isolated from the advantage of receiving more query information. Online receives plan roles, filter literals, and intermediate execution state, whereas Question-only receives the question, schema, and sampled values. The context study shows that revealing explicit literals alone improves semantic-error accuracy by 6.8 points. Therefore, the Online-versus-Question-only gap can reflect both richer information and interleaved execution. A query-aware non-interleaved control that receives the same plan-derived candidates and context would be needed to attribute the gain specifically to interleaving; otherwise the contribution should be framed as a combined query-aware policy.

Questions for authors
  1. In the intended deployment setting, how would the executable query plan, plan roles, and filter literals be obtained? Please clarify whether the paper’s main claim concerns error detection given a reference plan, rather than end-to-end TQA performance.

  2. In the generated-plan experiment, can the authors also report Offline under the same generated SQL used for Online and Hybrid? The point is to test whether the policy comparison remains favorable when query-generation errors affect all methods.

  3. Could the authors compare Online with a query-aware but non-interleaved policy that receives the same plan-derived candidate columns, filter literals, and detector context? This would isolate the effect of interleaving from the effect of receiving richer query information.

3 comments

We thank the reviewer for their constructive feedback. Below, we clarify the planning assumption and report additional comparisons. We will incorporate these clarifications and results into the revised manuscript.

W1/Q1

As stated on page 3, line 150, the main experiments assume a fixed executable reference query. An incorrect generated plan can misidentify the query scope and direct detection toward irrelevant data. Using ground-truth plans allows us to isolate data-error detection quality from these planning errors. Query planning is an important, well-studied problem with its own challenges yet our focus is on how to allocate a limited detection budget given a query plan. We will clarify that the paper focuses on error detection given a reference plan, rather than end-to-end TQA.

In deployment, an off-the-shelf NL2SQL planner can generate the query. Appendix F evaluates a planner given the question, schema and sampled values from corrupted data. To test whether our findings hold across planning models, we evaluated different planning LLMs on a sample of 160 question-seed instances:

Planning configurationOffline accuracyOnline accuracy
Reference plans52.5%73.8%
GPT-5-mini34.4%48.1%
GPT-4.1-mini20.6%29.4%
GPT-4o-mini20.0%28.8%

Planning errors reduce absolute answer accuracy and both policies remain sensitive to the quality of planning, still Online retains an advantage of 8.8–13.8 percentage points over Offline across the three tested planners.

W1/Q2

We extended the comparison in Appendix F, Table 24 to include all policies under the same settings.

PolicyAnswer accuracyTokens per question
No repair29.4%682
Offline33.3%6,455
Question-only44.4%4,538
Online42.6%3,915
Hybrid38.9%6,939

Online improves on Offline by 9.3 percentage points (28.1% relative) while using fewer tokens. Question-only has the highest answer accuracy in this setting for reasons already discussed in section 6.2.

W2/Q3

Thank you for suggesting this comparison. We conducted an additional controlled ablation that follows the same experimental set up from the paper (using a sample of 160 question-seed instances), but varies the error placement in these experiments. We place the errors in columns used to filter rows and columns used later to compute the answer. This placement tests a dependency between repairs: fixing a filtering column changes which rows contribute to the answer and which remaining errors matter.

The non-interleaved variant runs detection checks in a fixed order until the budget is exhausted, then repairs the flagged cells. The interleaved variant repairs after each detection action, examines how query execution changes, and uses that information to prioritize the next action. Both variants can inspect the same columns, but interleaving can use repair feedback to direct the remaining budget toward errors affecting the answer.

VariantAnswer accuracyMean detection tokens
Non-interleaved43.7%2,607
Interleaved82.5%2,710

At the default budget, interleaving improves answer accuracy by 38.7 percentage points with 3.9% more detection tokens. These results support using feedback from repairs to prioritize subsequent detection when repairs change which errors affect the answer. We thank the reviewer for this suggestion and will add it to the manuscript.

Thank you again for your feedback. We have addressed each of your questions through clarifications and additional experiments. Here is a short summary: We clarified the reference-plan assumption, showed that Online retains its advantage with generated plans across different planning LLMs, and added a controlled ablation demonstrating accuracy gains from interleaving with a small increase in token cost.

As today is the final day of discussion, we would appreciate your considering a higher rating if these responses resolve your concerns. Thanks again!

Thank you for clarifying the role of reference plans and for running the two comparisons I requested. Under the same generated SQL, Online improves answer accuracy over Offline from 33.3% to 42.6% while using fewer tokens per question, which addresses my main concern about whether this advantage survives planning errors. I also found the interleaving ablation informative: when repairing a filter changes which rows contribute to the answer, execution feedback substantially improves accuracy over a fixed detection order. These results address my main concerns, and I will raise my rating to weak accept. I encourage you to make the reference-plan assumption explicit alongside the headline result and to describe the interleaving gain in the context of the error placement tested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.