acceptodds
Under review as a conference paper at ICLR 2027

Understanding Errors in LLM-Based Question Answering over Imperfect Tables

Abstract

We investigate error discovery and handling in question answering over imperfect tables through controlled studies across three large language models (LLMs) on human-reviewed RADAR-T examples. Answering questions over these tables requires handling errors that can affect the answer. We vary row order and compare original, error-marked, and repaired tables to test whether discovery depends on where errors appear and whether providing their locations is sufficient for accurate question answering. First, reordering rows changes error discovery even when the table contents and gold answer remain unchanged. Complete discovery is higher for back than front placements and, averaged over the tested mean positions, for compact than widely spaced layouts. Second, providing verified error locations alone is insufficient for accurate QA, leaving a substantial accuracy gap between error-marked and repaired tables. Providing tables with human-reviewed repairs already applied raises code-assisted QA accuracy by 39.0–59.1 percentage points over the error-marked tables across the three systems. GBDI, a simple workflow, puts these findings into practice by combining error discovery across shuffled table views with explicit guidance for verifying and handling the reported errors. On RADAR-T, GBDI raises observed QA accuracy by 3.8–18.5 percentage points over a code-agent baseline across five systems. These results highlight the importance of both reliable error discovery and effective error handling in question answering over imperfect tables. Our anonymous repository is available at https://anonymous.4open.science/r/GBDI-ICLR-2027-85BD/

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, table question answering, imperfect tables, error discovery, error handling, LLM robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PxhUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies error discovery and handling in LLM-based question answering over imperfect tables. Using human-reviewed RADAR-T examples, the authors adjust the positions and spacing of relevant error rows while keeping the table contents and correct answers unchanged. They find that these changes affect whether models discover all relevant errors, with more pronounced effects in the Direct mode. Another set of experiments compares original tables, tables with verified errors marked, and tables with human-reviewed repair plans applied. The accuracy gap between the latter two conditions indicates that providing error locations alone is still insufficient to ensure reliable answering. Based on these findings, the authors propose GBDI: it independently inspects five random row permutations, merges candidate error reports, and uses explicit handling guidance to direct a code agent to verify and handle relevant errors before computing the answer. The controlled experiments involve three large language models, while the full QA evaluation covers five models and all 313 RADAR-T examples. Compared with the Code Agent baseline, GBDI improves accuracy by 3.8–18.5 percentage points, with paired 95% confidence intervals excluding zero for four systems.

Critical strengths

S1. The controlled experiments directly examine error discovery and subsequent handling. The authors vary the positions and spacing of critical error rows while keeping the table contents, questions, and correct answers unchanged, testing whether error discovery is affected by row arrangement. They also compare error-marked inputs with human-repaired inputs to examine the QA difficulties that remain after error locations are provided. These controls provide concrete evidence for understanding where models still fail and inform the subsequent workflow design. S2. The practical effectiveness of the workflow is supported by evaluation across models and component controls. Across five systems and 313 RADAR-T examples, GBDI improves accuracy over the Code Agent baseline by 3.8–18.5 percentage points, with paired 95% confidence intervals excluding zero for four systems. The authors also compare random row permutations with repeated inspections under the same call budget and compare different handling guidance while keeping the error ledger fixed. These results support the practical usefulness of the workflow across multiple systems, while the component controls help examine the additional effects of row arrangement and guidance.

Critical weaknesses

W1. The methodological novelty of GBDI is limited. Its main procedure is to inspect multiple shuffled row orders, merge candidate errors, and use manually written prompts to guide a code agent in processing the table. These choices follow from the preceding experimental findings and have practical value, but the additions mainly concern workflow organization and prompt design. The paper does not sufficiently explain what GBDI advances technically over existing approaches involving repeated inspection and task-oriented data cleaning. Thus, the paper contributes more through its analysis of the problem, while the novelty of the method itself is relatively limited. W2. The reliability of the critical decisions to modify or remove records is not sufficiently examined. GBDI retains candidates proposed by any inspection, including records nominated only once. The final agent therefore needs to distinguish actual errors from false positives before deciding whether to modify, remove, or retain a record. Although the paper provides detailed rules requiring evidence verification and checks of relationships between fields, these judgments and the checking code are still produced by the same LLM. Evaluation of the handling stage mainly considers whether the final answer is correct, without further examining whether the actual repairs are correct, whether valid values are unnecessarily changed, or whether valid records are mistakenly removed. For a paper emphasizing error handling, this stage needs more direct analysis to establish that the workflow improves the reliability of data processing.

Questions for authors

Q1. How were the rules in the Intervention Skill developed? Were they fixed before the final evaluation? Were RADAR-T evaluation examples or human-reviewed repair plans used when developing or revising them? Q2. GBDI retains candidate errors nominated by only one inspection. For false-positive nominations, does the final agent make unnecessary changes to valid data or remove valid records? Q3. Table 13 shows that many questions are still answered incorrectly even after all critical rows have been discovered. Where do these failures mainly occur: incorrect data repairs, or subsequent computation or code execution errors? Q4. Beyond the number of calls, how much additional overhead does GBDI incur compared with the standard Code Agent?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/3]

We sincerely thank the reviewer for the careful summary and for recognizing that our controlled experiments directly examine error discovery and handling, and that the effectiveness of the workflow is supported across models and component controls. To address the concerns, we carried out additional analyses and updated the manuscript accordingly (changes are highlighted in blue). Each response starts with a TLDR, followed by the details.

R to W1: Methodological novelty of GBDI

"The paper does not sufficiently explain what GBDI advances technically over existing approaches involving repeated inspection and task-oriented data cleaning. Thus, the paper contributes more through its analysis of the problem, while the novelty of the method itself is relatively limited."

Thank you for the suggestion to clarify what GBDI adds beyond existing approaches.

TLDR. Each GBDI component targets a failure mechanism found in our analysis and is tested against the corresponding existing approach. GBDI beats repeated inspection at the same budget and makes record-level decisions that column-level task-oriented cleaning cannot express.

1. The analysis behind GBDI is new. Our study asks whether an LLM finds and correctly handles the errors that affect a specific answer. Order-sensitivity studies on clean tables and answer-level benchmarks do not examine this question. With a human-reviewed reference of 154 instances, 1,162 critical rows and reviewed EDIT or DROP decisions, we identify two failure mechanisms. First, discovery depends on where the same errors appear. Back placements raise complete discovery by 11.6–16.7 points, and dispersed layouts lower it by 7.9–16.2 points. Second, handling is the larger bottleneck. Supplying repaired tables raises Code Agent accuracy to 92.9–98.1%, whereas marking the same errors changes it by at most 11.0 points.

2. What GBDI advances over repeated inspection. Repeated inspection reduces random noise but keeps every row in the same place, so an error that is hard to see at its position stays hard to see. GBDI makes three changes and tests each of them.

  • Permuted instead of repeated views. At the same five-call budget, random permutation raises Qwen's complete discovery from 58.9% to 78.7% (+19.9 points, 95% CI [11.1, 29.2]) and its correct answers from 190 to 203. The QA gain also holds for GLM (151 to 155) and MiniMax (145 to 160).
  • Union instead of consensus. Keeping only the rows that a majority of views nominate lowers complete discovery by 6–37 points across the five systems without a consistent gain in precision (Appendix D.1 of the revised manuscript).
  • Verification instead of filtering. Candidates are ordered by the number of supporting views and verified against the table, and the agent leaves 89–98% of the false nominations made by a single view untouched.

3. What GBDI advances over task-oriented data cleaning. Task-oriented cleaning such as AutoDCWorkflow solves a different task. It outputs a cleaned table through column-level operations, whereas GBDI answers the question and must decide for each erroneous record whether its value can be recovered from the table or the record must be excluded. In 41.7% of the 1,180 reviewed repair actions the record has to be excluded, which column-level operations cannot do. The Intervention Skill encodes these record-level decisions per error type, and its contribution is measured directly.

Qwen Plus, Random-5 ledgerCorrect answers of 313
Generic guidance183
Full Skill203
Skill without value-recovery rules186
Skill without formatting rules190
Skill without execution checks197

With the same ledger, the Skill also improves accuracy for GLM (144 to 155 correct answers) and MiniMax (140 to 160), which were not used to develop it.

We have stated these points in the new Discussion section of the revised manuscript.

AuthorsOpenReview ↗

Rebuttal by Authors [2/3]

R to W2: Reliability of the decisions to modify or remove records

"Although the paper provides detailed rules requiring evidence verification and checks of relationships between fields, these judgments and the checking code are still produced by the same LLM. Evaluation of the handling stage mainly considers whether the final answer is correct, without further examining whether the actual repairs are correct, whether valid values are unnecessarily changed, or whether valid records are mistakenly removed."

Thank you for the suggestion to evaluate the handling stage directly.

TLDR. Checked against the independent human-reviewed references, the changes the agent makes are mostly correct. Edited values are rarely wrong, valid data are rarely changed, and the main failure is leaving a critical row unrepaired rather than making a harmful change. The Skill also repairs better than generic guidance.

1. Are the repairs correct? We re-executed the recorded code of every trajectory on the original table to recover the table on which the agent computed its answer (1,561 of the 1,565 trajectories are reproduced exactly), and compared it with the reviewed actions and values on the 154 reviewed instances. The results of this and the next two analyses are reported in Appendix D.7 of the revised manuscript.

Critical rows repaired exactly, %EDIT rows with a wrong value, %Accuracy when all critical rows are exact, %
Qwen Plus57.77.798.3
GLM-4.5-Air46.013.082.2
MiniMax-M2.521.0†21.965.5
GPT-5 Mini52.08.596.6
DeepSeek-V4.1-Flash52.27.0100

† MiniMax often corrects values only inside its answer code without saving them to the table, so this is a lower bound.

2. Are valid values changed or valid records removed? We treat rows that RADAR did not perturb as valid and examine all 313 instances.

Single-view false nominations changed, %Multi-view false nominations changed, %Error-free tables with any changeAccuracy on error-free tables
Qwen Plus3.926.89/5349/53
GLM-4.5-Air1.820.95/5348/53
MiniMax-M2.54.724.40/5350/53
GPT-5 Mini11.144.45/5349/53
DeepSeek-V4.1-Flash7.742.70/5352/53

False nominations from a single view are almost always left untouched. Rows nominated by several views are changed more often. Since unperturbed rows may still contain problems in the source data, these rates are upper bounds on unnecessary changes. On the 53 error-free tables, accuracy stays at 48–52.

3. Does the guidance itself improve the repairs? We held the candidate list fixed and replaced the Skill with generic guidance.

Same Random-5 ledgerRows repaired exactly, % (Generic → Skill)EDIT rows with the exact value, % (Generic → Skill)
Qwen Plus55.9 → 57.745.6 → 56.0
GLM-4.5-Air43.5 → 46.048.4 → 55.1
MiniMax-M2.513.5 → 21.015.8 → 22.8

The Skill improves exact repairs for all three systems, mainly by recovering the correct value. For Qwen it also halves the changes to rows without injected errors, from 4.8% to 2.3% of such rows.

4. Verification by the same model. The reference above is independent of the model, so points 1–3 measure the quality of its self-verified decisions directly. The changes it makes are mostly correct, and only 3 of the 203 wrong answers after complete discovery follow changes to valid rows (see Q3). Its main failure is leaving critical rows unrepaired, not modifying or removing valid data. In addition, every action must cite evidence visible in the table, is recorded in an intervention log and is executed as code on a copy of the table, so each change can be audited.

R to Q1: Development of the Intervention Skill

"How were the rules in the Intervention Skill developed? Were they fixed before the final evaluation? Were RADAR-T evaluation examples or human-reviewed repair plans used when developing or revising them?"

Thank you for this question.

The rules target the general task rather than individual instances. They state when a value may be corrected (EDIT), excluded (DROP) or kept (NO-OP), add recovery rules for missing values, bad values and formatting, and contain no table, column or value-specific content (Appendix F.5). We developed them with Qwen Plus on a small subset of RADAR-T. The Skill was frozen after development, and the human-reviewed repair plans of Section 2 were completed after the freeze and were never used to develop or revise the rules.

AuthorsOpenReview ↗

Rebuttal by Authors [3/3]

R to Q2: Unnecessary changes from single-view nominations

"GBDI retains candidate errors nominated by only one inspection. For false-positive nominations, does the final agent make unnecessary changes to valid data or remove valid records?"

Thank you for the question.

TLDR. Rarely. The agent leaves 89–98% of single-view false nominations untouched, and accuracy on error-free tables stays at 48–52 of 53.

We treat rows that RADAR did not perturb as valid. Across the five systems, only 1.8–11.1% of single-view false nominations are changed (Appendix D.7). False nominations repeated across views are changed more often (21–44%), and since unperturbed rows may still contain problems in the source data, this rate is an upper bound. The number of supporting views is informative as well, since the share of truly erroneous rows rises from 23–60% among single-view nominations to 78–100% among rows nominated by all five views. This is why GBDI keeps single-view candidates for verification but orders the list by support.

R to Q3: Failures after complete discovery

"Table 13 shows that many questions are still answered incorrectly even after all critical rows have been discovered. Where do these failures mainly occur: incorrect data repairs, or subsequent computation or code execution errors?"

Thank you for the question.

TLDR. Mainly in the repairs. Of the 203 wrong answers after complete discovery, 92% involve a critical row not repaired as reviewed, and only 6.4% follow a fully correct repair.

We assign each wrong answer to the first stage that fails. This breakdown refines Table 13 and has been added to Appendix D.7 of the revised manuscript. This result is also coincide with our conclusion that within the repairs, choosing the action is harder than carrying it out.

Wrong answers after complete discoveryQwenGLMMiniMaxGPT-5 MiniDeepSeekTotal
A critical row not repaired as reviewed3948344026187 (92.1%)
Valid rows changed030003 (1.5%)
Wrong after an exact repair (computation or code error)0382013 (6.4%)

R to Q4: Overhead beyond the number of calls

"Beyond the number of calls, how much additional overhead does GBDI incur compared with the standard Code Agent?"

Thank you for the question.

TLDR. GBDI in 5 views uses 2.6–4.0 times the Code Agent's tokens, about half on discovery. Verification adds little. However, fewer views can reduce the cost.

Tokens per instanceDirectCode AgentGBDI (5 views)GBDI's discovery part
Qwen Plus23.8k27.1k90.4k45.8k
MiniMax-M2.530.9k36.3k94.5k43.4k
GPT-5 Mini8.1k20.7k83.2k44.6k

Each discovery view costs about 7–9k tokens. Verifying a longer candidate list adds little because all candidates are checked within one turn. For Qwen, the answering stage uses 42.2k tokens without a list and 44.6k with the five-view list. Since the gain in discovery diminishes after two or three views, using fewer views is a reasonable way to reduce the overhead. We have added the token comparison to Appendix D.8 of the revised manuscript.

We sincerely hope that these clarifications and new results address your concerns, and we would be glad to discuss any remaining questions. Thank you again for your time and valuable feedback.

Reviewer_1Dc4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper offers useful insights into error discovery and handling in LLM-based table question answering (TQA) over imperfect tables. Its controlled studies reveal sensitivity to row placement and a substantial accuracy gap between error-marked and repaired tables. Based on these findings, the authors propose GBDI, a workflow that aggregates candidate errors across shuffled table views and applies explicit guidance for verifying and handling errors before answering. However, the evaluation is concentrated on a single benchmark, and the analysis does not fully characterize the failure modes or efficiency trade-offs of the proposed mitigation. I see potential for a stronger contribution through broader evaluation and a more systematic decomposition of discovery, repair, and downstream reasoning failures, but the current study leaves these aspects insufficiently established.

Critical strengths
  1. Clear and timely research question. The paper distinguishes between discovering question-relevant errors and handling them to produce correct answers, addressing an important gap in TQA research that often assumes clean inputs.
  2. Controlled analysis of error discovery. Findings indicate error discovery itself is sensitive to placement of errors.
  3. Evidence that discovery alone is insufficient. The comparison of original, error-marked, and human-repaired tables reveals a substantial gap between identifying errors and resolving them. It motivates the need for an explicit repair strategy to match the upper bound (w/ human-reviewed repairs).
  4. Practical mitigation strategy. GBDI addresses placement sensitivity by combining discovery across shuffled table views with guided intervention.
Critical weaknesses
  1. Limited empirical scope and generalizability. The main evaluation is concentrated on a subset of RADAR-T. Needs evaluation on more benchmarks to establish the generalizability of the findings. It is also unclear how the findings would generalize to a broader set of error types, query complexity, table sizes, query footprint.
  2. Insufficient characterization of GBDI’s efficiency and trade-offs. Although the paper includes useful controls, including repeated inspections at a matched call budget and intervention-guidance ablations, it does not sufficiently characterize the cost–benefit trade-off of the overall workflow. Aggregating nominations across shuffled views may improve error coverage but also increase duplicate or false-positive nominations and downstream verification effort. Reporting token and execution costs alongside nomination precision, redundancy, and final QA gains would better establish when GBDI is worthwhile.
  3. Incomplete decomposition of downstream failure modes. The distinction between error discovery and handling is valuable, but the gap between error-marked and repaired tables does not, by itself, isolate the specific sources of downstream failure. These may include selecting an appropriate repair or row-exclusion action, executing the intervention correctly, and reasoning over the resulting table.
  4. Missing baselines and ablation experiments for GBDI. The multi-view strategy improves the robustness of error discovery, but it is unclear whether repeatedly inspecting the full table is the only and most efficient way to address missed errors. Comparing GBDI with alternative strategies for handling errors would be useful.
  5. Complete discovery rate captures whether all answer-critical rows are found, but it does not penalize unnecessary nominations. This can become critical if error handling is expensive and budgeted.
Questions for authors
  1. GBDI addresses presentation sensitivity through repeated inspections of shuffled table views, what is the marginal yield of each view? Since each view would require additional inference calls, the costs can quickly add up, resulting in an accuracy-cost trade-off.
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/3]

We sincerely thank the reviewer for recognizing the clear and timely research question, the controlled analysis of error discovery, and the evidence that discovery alone is insufficient. To address the concerns, we carried out additional analyses and updated the manuscript accordingly (changes are highlighted in blue). Since the weaknesses in the review are not numbered, we refer to each of them by its topic. Each response starts with a TLDR, followed by the details.

R to scope and generalizability

"The main evaluation is concentrated on a subset of RADAR-T. Needs evaluation on more benchmarks to establish the generalizability of the findings. It is also unclear how the findings would generalize to a broader set of error types, query complexity, table sizes, query footprint."

Thank you for the suggestion to clarify the scope of our evaluation.

TLDR. The end-to-end evaluation covers all of RADAR-T, to our knowledge the only benchmark for QA over imperfect tables. The discovery effects also hold on WikiSQL tables, and GBDI's gain grows with table length and is positive in 14 of the 15 groups by query footprint.

1. The main evaluation covers all of RADAR-T. The end-to-end evaluation of GBDI uses all 313 RADAR-T instances. They come from 53 tasks over 27 source datasets in domains such as health, transport, sports, finance, environment and HR, and they cover all five error types as well as the error-free tables (Appendix A.4). Only the controlled diagnostics in Sections 3 and 4 use the human-reviewed subset, because they need answer-preserving manipulations and a verified reference.

2. The discovery effects hold beyond RADAR-T. We constructed missingness instances from 41 WikiSQL tables and evaluated two further models, Qwen-Turbo and Qwen3-8B (Appendix C.3). Spreading the critical rows apart lowers complete discovery from 80.5% to 48.0% for Qwen-Turbo and from 64.2% to 35.0% for Qwen3-8B, and complete discovery is higher at the back of the table than at the front for both models.

3. Table size and query footprint. Following your suggestion, we stratified the gain of GBDI over the Code Agent by table length and by the number of critical rows that the question touches.

Gain over Code Agent (pp)≤ 100 rows101–160 rows> 160 rows1–4 critical rows5–9 critical rows≥ 10 critical rows
Instances5417089524841
Qwen Plus+0.0+10.6+18.0+9.6+18.8+29.3
GLM-4.5-Air+1.9+11.8+5.6+7.7+10.4+4.9
MiniMax-M2.5+14.8+18.8+20.2+19.2+22.9+24.4
GPT-5 Mini−5.6+2.9+11.2+17.3−2.1+12.2
DeepSeek-V4.1-Flash+1.9+7.6+12.4+19.2+14.6+7.3

The gain is larger on long tables than on short tables for every system. This is expected, because position and spacing effects matter most when the table is long. Across the number of critical rows, the gain is positive in 14 of the 15 system–group cells.

4. Error types. The table below reports the share of critical rows repaired exactly as reviewed, by error type.

Exact repairs, %Bad valuesLogical inconsistencyFormattingMissing valuesOutliers
Qwen Plus45.657.362.849.270.8
GLM-4.5-Air53.111.982.926.719.3
MiniMax-M2.519.57.628.518.324.0
GPT-5 Mini40.737.869.841.455.2
DeepSeek-V4.1-Flash54.031.466.347.647.4

Formatting errors are the easiest to repair on average, while logical inconsistencies, which require checking several fields of a record against each other, are the hardest for four of the five systems.

AuthorsOpenReview ↗

Rebuttal by Authors [2/3]

R to cost–benefit trade-off

"Aggregating nominations across shuffled views may improve error coverage but also increase duplicate or false-positive nominations and downstream verification effort. Reporting token and execution costs alongside nomination precision, redundancy, and final QA gains would better establish when GBDI is worthwhile."

Thank you for the suggestion.

TLDR. GBDI uses 2.6–4.0 times the Code Agent's tokens, about half of it on discovery. Merging views removes most duplicates, precision does not drop as views are added, and false positives rarely cause changes. GBDI pays off most on long tables, and using fewer views cuts the cost.

1. Token cost. The table gives the average tokens per instance over all 313 instances, taken from the providers' usage records (Appendix D.8).

Tokens per instanceDirectCode AgentGBDIof which discovery
Qwen Plus23.8k27.1k90.4k45.8k
MiniMax-M2.530.9k36.3k94.5k43.4k
GPT-5 Mini8.1k20.7k83.2k44.6k

2. Redundancy and precision. Merging five views removes most duplicate nominations, and precision stays the same or rises as views are added.

Raw nominations per instanceUnique rows per instancePrecision with 1 view, %Precision with 5 views, %
Qwen Plus30.19.671.872.5
GLM-4.5-Air26.413.839.159.7
MiniMax-M2.513.45.736.065.1
GPT-5 Mini23.28.082.282.4
DeepSeek-V4.1-Flash26.46.370.274.7

3. Verification effort and false positives. All candidates are verified within a single reserved turn, so the verification cost grows only slowly with the length of the list. For Qwen, the answering stage uses 42.2k tokens without a candidate list and 44.6k with the five-view list. False positives also rarely lead to changes, since the agent leaves 89–98% of the rows nominated by only one view untouched (Appendix D.7 of the revised manuscript).

4. Benefit relative to cost. For Qwen, accuracy rises steadily with the number of views (Table 15 of the original paper).

Qwen PlusTokens per instanceQA accuracy, %
Code Agent27k54.0
Skill without discovery42k57.5
GBDI with 1 viewabout 51k59.7
GBDI with 3 viewsabout 70k61.0
GBDI with 5 views90k64.9
5 repeated views without permutationabout 90k60.7

At the same budget, permuted views reach 64.9% while repeated views reach 60.7%, so the gain comes from the permutation rather than from the extra calls. The stratified gains in our response on scope add that GBDI pays off most on long tables and on questions that touch many critical rows. Under a tight budget, fewer views are a reasonable choice (see our responses on alternatives and on marginal yield).

R to sources of downstream failure

"The gap between error-marked and repaired tables does not, by itself, isolate the specific sources of downstream failure. These may include selecting an appropriate repair or row-exclusion action, executing the intervention correctly, and reasoning over the resulting table."

Thank you for this valuable suggestion.

TLDR. Choosing the action is the main source of error, while execution mostly succeeds and reasoning over a correctly repaired table rarely fails.

We re-executed the recorded code of every GBDI trajectory on the original table to recover the table on which the agent computed its answer (1,561 of the 1,565 trajectories are reproduced exactly). We compared this table with the human-reviewed actions and values, and we also compared the decision the agent wrote into its own intervention log with the reviewed action.

Below are the Results for the 3 stages:

Selecting, logged decision agrees with the review, %Executing, logged decision applied to the table, %Reasoning, accuracy when all critical rows are repaired exactly, %
Qwen Plus81.583.898.3
GLM-4.5-Air62.583.982.2
MiniMax-M2.564.292.665.5
GPT-5 Mini75.591.196.6
DeepSeek-V4.1-Flash75.795.4100

Among the instances in which every critical row was discovered, there are 203 wrong answers across the five systems. We assign each of them to the first stage that fails. This breakdown refines Table 13 and has been added to Appendix D.7 of the revised manuscript.

Wrong answers after complete discoveryQwenGLMMiniMaxGPT-5 MiniDeepSeekTotal
A critical row not repaired as reviewed3948344026187 (92.1%)
Valid rows changed030003 (1.5%)
Wrong after an exact repair0382013 (6.4%)

The main remaining difficulty is therefore the choice between correcting and excluding a record. This agrees with the Skill ablation in Table 12 of the original paper.

AuthorsOpenReview ↗

Rebuttal by Authors [3/3]

R to alternatives to re-inspecting the full table

"The multi-view strategy improves the robustness of error discovery, but it is unclear whether repeatedly inspecting the full table is the only and most efficient way to address missed errors. Comparing GBDI with alternative strategies for handling errors would be useful."

Thank you for the suggestion to compare GBDI with alternative strategies.

TLDR. Under the same budget, random permutation with a union list beats repeated inspection, cyclic shifts, majority voting, fewer views and no discovery.

1. Alternatives under the same budget.

Correct answers of 313No discoveryRepeated-5Cyclic-5Random-5 (GBDI)
Qwen Plus180190195203

Keeping only the rows nominated by a majority of views lowers complete discovery by 6–37 points (Appendix D.1 of the revised manuscript; see also our response to Reviewer yMoc, W1.2). Using fewer views lowers both discovery and accuracy (see the response on marginal yield below).

2. Why the full table is inspected. Critical rows are spread over most of the table, with a median distance of 78% of the table length between the first and the last critical row of an instance. Inspecting only part of the table in each call would therefore still require covering all rows, and it would add the risk of separating rows that must be compared with each other, such as the rows of a logical inconsistency.

R to CDR and unnecessary nominations

"Complete discovery rate captures whether all answer-critical rows are found, but it does not penalize unnecessary nominations. This can become critical if error handling is expensive and budgeted."

Thank you for raising this point about the evaluation metric.

TLDR. A missed critical row costs far more than an extra nomination, so CDR is the primary metric, but the paper also reports precision and false-positive rates. Under a handling budget, the support ranking lets a system verify the most reliable candidates first.

1. Why CDR is the primary metric. A single missed critical row can make the answer wrong, whereas an extra nomination is verified against the table and is usually left untouched. In GBDI, all candidates are verified in one turn, and for Qwen the answering stage uses 44.6k tokens with the five-view list compared with 42.2k without any list.

2. Unnecessary nominations are reported as well. The paper reports precision, recall, F1, the mean number of nominated rows and the false-positive rate on error-free instances next to CDR (Appendix B.6 and Table 9). With five views, 60–82% of the nominated rows are critical, and precision does not drop as views are added (see the response on cost–benefit).

3. If handling were expensive and budgeted. The number of supporting views is a reliable priority signal. The share of truly erroneous rows rises from 23–60% among rows nominated by one view to 78–100% among rows nominated by all five views (Appendix D.1 of the revised manuscript). A budgeted system could therefore verify candidates in order of support and stop when the budget is spent, which keeps the most reliable candidates while bounding the handling cost.

R to marginal yield of each view

"GBDI addresses presentation sensitivity through repeated inspections of shuffled table views, what is the marginal yield of each view?"

Thank you for the question.

TLDR. Each extra view raises complete discovery, by 6–20 points for the second view and at most 4.1 for the fifth, at about 7–9k tokens per view. Two or three views suit a tight budget.

The table gives complete discovery by number of views, averaged over all subsets of the five views of each size.

Complete discovery (%) by number of views12345
Qwen Plus45.565.973.576.778.7
GLM-4.5-Air32.347.656.462.666.7
MiniMax-M2.523.738.848.154.258.2
GPT-5 Mini50.963.369.673.877.3
DeepSeek-V4.1-Flash66.772.674.575.375.9

Complete discovery increases with every added view, but the gain becomes smaller. The second view adds 6–20 points, while the fifth adds at most 4.1 points. For Qwen, final accuracy follows the same pattern, rising from 57.5% without discovery to 59.7% with one view, 61.0% with three views and 64.9% with five views (Table 15 of the original paper). When the budget is limited, two or three views therefore offer a good trade-off between discovery and cost.

We sincerely hope that these clarifications and new results address your concerns, and we would be glad to discuss any remaining questions. Thank you again for your time and valuable feedback.

Reviewer_yMocOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study why LLM-based table question answering fails on imperfect tables, separating error discovery from error handling. They annotate 154 RADAR-T instances with critical cells, which are erroneous and relevant to the question, and with reviewed EDIT or DROP repairs. The study consists of three parts: (1) answer-preserving row permutations that move critical rows across positions and spacings to measure the complete discovery rate (CDR), (2) a comparison of QA accuracy on raw, error-marked, and repaired tables, and (3) GBDI, a workflow that merges error reports from five shuffled table views and lets a code agent repair the table under a detailed prompt called the Intervention Skill. The authors report that discovery is higher when critical rows appear near the back of the table and close together, and that repaired tables beat error-marked tables by 39.0 to 59.1 points with the code agent. On all 313 RADAR-T instances, GBDI is reported to improve accuracy over the code agent baseline by 3.8 to 18.5 points across five LLMs.

Critical strengths

S1. The authors tackle an important problem in imperfect-table QA by separating error discovery from error handling, supported by a human-reviewed reference of critical cells and repairs.

Critical weaknesses

W1. Insufficient novelty and justification of key contributions

The error-discovery analysis and GBDI are presented as the main contributions, but their novelty over existing work on table order sensitivity, imperfect-table QA, and data cleaning is unclear.

W1.1. The findings largely extend known results. Liu et al. [1] and FREB-TQA [2] already show that model answers change when the same table content is reordered or the answer-containing rows are moved, and RADAR [3] already evaluates how LLMs answer questions over tables with injected errors. The paper measures error discovery instead of final answers, but does not explain what this distinction reveals beyond [1], [2], and [3]. In addition, AutoDCWorkflow [4] already uses an LLM to select columns relevant to a given question, assess their quality, and apply cleaning operations, yet the paper does not explain how GBDI differs from it. The authors should clarify the new insight and the distinction from [4].

W1.2. The discovery step of GBDI is not clearly distinct from existing methods. It shuffles rows uniformly at random and aggregates the outputs (Section 5.1), much like permutation self-consistency [5], which the paper does not cite. The authors should clarify how GBDI differs from [5]. Since Section 3 finds higher discovery for back placements, one option is to build on this finding, for example with cyclic shifts that place every row near the back of at least one view.

[1] Liu, Tianyang, Fei Wang, and Muhao Chen. "Rethinking tabular data understanding with large language models." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

[2] Zhou, Wei, et al. "FREB-TQA: A fine-grained robustness evaluation benchmark for table question answering." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

[3] Gu, Ken, et al. "RADAR: Benchmarking Language Models on Imperfect Tabular Data." Advances in Neural Information Processing Systems 38 (2026).

[4] Li, Lan, et al. "AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark." EMNLP (Findings). 2025.

[5] Tang, Raphael, et al. "Found in the middle: Permutation self-consistency improves listwise ranking in large language models." Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024.

W2. Insufficient experimental validation

The experiments neither compare GBDI with strong baselines nor evaluate the quality of its interventions.

W2.1. The baselines are insufficient. GBDI is compared only with Direct and Code Agent, and its largest gain (MiniMax-M2.5) is measured against a Code Agent that scores below Direct (Table 20). The authors should compare against stronger baselines, such as AutoDCWorkflow [4].

W2.2. Intervention quality is not evaluated. The paper does not measure whether GBDI chooses the correct EDIT or DROP action and replacement value. The authors should report this accuracy against the human-reviewed references (Section 2).

Questions for authors

Please refer to W1 and W2.

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors [1/4]

We sincerely thank the reviewer for the careful reading and the constructive suggestions. To address the concerns, we carried out additional experiments and analyses and updated the manuscript accordingly (changes are highlighted in blue). Each response starts with a TLDR, followed by the details.

R to W1.1: What measuring discovery adds beyond [1]–[3], and the difference from AutoDCWorkflow [4]

"The paper measures error discovery instead of final answers, but does not explain what this distinction reveals beyond [1], [2], and [3]. … The authors should clarify the new insight and the distinction from [4]."

Thank you for the opportunity to clarify our contribution and its relation to these works.

TLDR. We measure a step that [1]–[3] do not isolate, namely whether an LLM finds and correctly handles the errors behind a specific answer. This shows that discovery depends on the table layout and that handling is the larger bottleneck. AutoDCWorkflow [4] solves a different task, data cleaning rather than end-to-end table QA.

1. The problem we study is overlooked by [1]–[3]. [1] and [2] reorder clean tables or move the answer rows and measure whether the final answer changes. Their evidence rows hold correct values, so the model only has to locate and read them. RADAR [3] provides imperfect tables and reports the accuracy drop. None of them asks whether the model noticed the erroneous values that affect this particular answer and handled each of them correctly. We formalize these values as critical cells, i.e., cells that are both erroneous and relevant to the question.

2. The errors are harder than reading correct evidence. Our errors come from RADAR [3], a recent NeurIPS benchmark for imperfect tables, and cover five types found in real data (missing values, bad values, outliers, formatting errors and logical inconsistencies). A nonempty instance has a median of 6 critical rows, and the first and last of them lie a median of 78% of the table length apart.

3. The new insight concerns where LLMs fail. Scoring only the final answer, [1]–[3] cannot tell an error that was never found from one that was found but mishandled. Separating the two steps yields two findings.

  • Discovery depends on presentation. With the question, cell values and gold answer fixed, moving the same critical rows from the front to the back raises complete discovery by 11.6–16.7 points under Direct, and spreading them apart lowers it by 7.9–16.2 points (Section 3). The spacing effect involves several error rows at once, which [1] and [2] do not vary.
  • Handling is the larger bottleneck. Marking the verified errors changes Code Agent accuracy by at most 11.0 points, whereas supplying the repaired table raises it to 92.9–98.1% (Section 4). Once the errors are handled correctly, the remaining computation is rarely the problem.

These findings motivate the two components of GBDI, namely shuffled views for discovery and explicit guidance for handling.

4. Difference from AutoDCWorkflow [4]. [4] and our work solve different tasks. [4] is a data-cleaning system. Given a dirty table and an analysis purpose, it outputs a cleaned table and a reusable OpenRefine workflow, and it is evaluated against a human-cleaned table and workflow (Sec. 5 of [4]). The purpose only guides which columns to clean, and answers are obtained by querying the cleaned table afterwards. Our task is end-to-end question answering over an imperfect table. The LLM receives the table and the question and must return the answer, so it has to notice the errors that affect this particular answer and handle them while reasoning. Our paper studies where LLMs fail in this process, and GBDI is a QA workflow built on these findings, not a cleaning tool. The 2 tasks also involve different errors.

  • a) [4] fixes how values are written. Its errors are spelling variants, inconsistent case or spacing, and strings in numeric columns (Sec. 5.1.1). They are frequent, a column sample reveals them, and a column-wide operation fixes them without locating any record.

  • b) Our errors are wrong values in a few records. They look like ordinary data, so the model must find them among all rows and then correct each with evidence from other fields or rows or exclude it. Of the 1,180 human-reviewed repair actions, 41.7% exclude a record and 16.0% use other fields or rows, while at most 42.3% could be produced by the operations of [4].

We have added these distinctions to the new Discussion section.

AuthorsOpenReview ↗

Rebuttal by Authors [2/4]

R to W1.2: Relation to permutation self-consistency [5] and cyclic shifts

"It shuffles rows uniformly at random and aggregates the outputs (Section 5.1), much like permutation self-consistency [5] … one option is to build on this finding, for example with cyclic shifts that place every row near the back of at least one view."

Thank you for pointing us to [5] and for the concrete suggestion. We now cite [5] and discuss it in the revised manuscript.

TLDR. PSC targets listwise ranking tasks such as passage reranking and does not consider tables. We transfer its permutation idea to tables and replace consensus with union+skill verification. The cyclic shift you suggested discovers as much as random permutation but answers fewer questions comparing to our method.

1. What PSC addresses, and what changes when it is transferred to tables. PSC was designed for listwise ranking. It shuffles the order of the candidate passages in the prompt, obtains one ranking per order, and merges these rankings into a consensus. In that setting every output is a complete ranking of the same passages, and there is no external evidence to tell which ranking is right, so consensus is the natural way to cancel position bias.

PSC also does not consider tables. Transferring its idea to error discovery in tables changes two things. First, each view returns only the rows it finds suspicious, and because of the position and spacing effects in Section 3, a critical row may be found only when it appears in a favorable position. A consensus rule removes exactly these rows, and missing one critical row can change the answer. Second, the table itself is evidence, so every candidate can be checked against it, which makes a false nomination cheap.

2. Our adaptation. Based on these two differences, GBDI keeps PSC's permutation of the input but replaces consensus with the union of all views' candidates, ordered by the number of views that nominate them, and requires the agent to verify each candidate against the table before changing anything. To test this adaptation, we applied a PSC-style rule to the same 5 views, keeping only the rows nominated by at least 3 views. Majority voting lowers complete discovery by 6–37 points without a consistent gain in precision.

QwenGLMMiniMaxGPT-5 MiniDeepSeek
Complete discovery, union (ours), %78.766.758.277.375.9
Complete discovery, majority vote, %61.031.921.357.470.2
Precision, union / majority vote, %72.5 / 71.959.7 / 43.065.1 / 36.082.4 / 84.074.7 / 69.9

The rows that the union keeps and voting drops are rarely harmful. The agent leaves 89–98% of the false nominations made by a single view untouched. The number of supporting views is still informative, since the share of truly erroneous rows rises from 23–60% among single-view nominations to 78–100% among rows nominated by all five views. GBDI therefore uses support to order the candidates rather than to filter them. For Qwen, this support sweep is also reported in Table 14(b) of the original paper. We have added the majority-vote comparison and this support analysis for all five systems to Appendix D.1 of the revised manuscript.

3. Cyclic shifts, following your suggestion. We evaluated Cyclic-5, which splits the table into five contiguous blocks and rotates them so that every row lies near the end of one view. Repeated-5, Cyclic-5 and Random-5 share the same five-call budget, discovery prompt and Intervention Skill.

Complete discovery, % (Repeated / Cyclic / Random)Correct answers of 313 (Repeated / Cyclic / Random)
Qwen Plus58.9 / 76.6 / 78.7190 / 195 / 203
GLM-4.5-Air61.0 / 73.8 / 66.7151 / 151 / 155
MiniMax-M2.538.3 / 60.3 / 58.2145 / 148 / 160

The results support the position finding of Section 3 in our paper. Placing every row near the end of one view raises complete discovery well above Repeated-5, to a level close to Random-5. The gain in discovery, however, does not carry over to final accuracy, and Cyclic-5 answers fewer questions correctly than Random-5 for all three systems.

We have added this comparison to Appendix D.1 of the revised manuscript and keep random permutation as the default.

AuthorsOpenReview ↗

Rebuttal by Authors [3/4]

R to W2.1: Baselines

"GBDI is compared only with Direct and Code Agent, and its largest gain (MiniMax-M2.5) is measured against a Code Agent that scores below Direct (Table 20). The authors should compare against stronger baselines, such as AutoDCWorkflow [4]."

Thank you for this suggestion. We respectfully note that GBDI is not compared only with Direct and Code Agent. The original paper also compares it with Repeated-5 at the same call budget (Section 5.2), the same agent without discovery (M=0, Table 15) and generic guidance on the same candidate list (Appendix D.2). Below we explain why they are sufficient.

TLDR. We run every approach that could give a stronger baseline (code-based answering, cleaning before answering, extra calls against order sensitivity) with the same model and inputs as GBDI. GBDI beats all of them. And no published method addresses exactly this task before.

1. Which baselines could be stronger. No published method finds and handles the errors that affect a question in TQA, so stronger baselines must come from three neighboring approaches. The first answers with code that can inspect and clean the table. The second cleans the table first and then answers, as in task-oriented data cleaning. The third spends extra calls on the same table to reduce order sensitivity, as in self-consistency or PSC. We implemented each one as a control with the same model, tables, questions, answering protocol and scorer as GBDI, so differences come from the approach itself and not from its implementation.

2. Why this set is sufficient. First, no published method addresses this task. TQA methods such as StructGPT and Chain-of-Table assume clean tables and have no step for finding or repairing errors. AutoDCWorkflow [4] solves a different task (as we mentioned before in W1.1) and outputs a cleaned table rather than an answer. Our M=0 control already covers this kind of approach with a larger action space, since the agent may edit any cell or exclude any record before answering, and it still answers 23 fewer questions than GBDI.

Second, we follow the evaluation setting of RADAR [3], the benchmark that defines QA over imperfect tables. RADAR evaluates LLMs with Direct and Code Agent, and we run both with its prompts, settings and scorer, so our results are directly comparable with the benchmark. The Code Agent is not a weak baseline. It can write any checking and cleaning code over the full table, and it outperforms Direct for four of the five systems.

Third, on top of RADAR's protocols, we run each neighboring approach as a matched control. The table gives correct answers out of 313 runs.

ApproachControlQwenGLMMiniMax
Benchmark protocolsDirect3714139
Code Agent169129102
Extra callsRepeated-5 + Skill190151145
Cyclic-5 + Skill195151148
Component checkRandom-5 + generic guidance183144140
OursGBDI (Random-5 + Skill)203155160

Across all 5 systems, GBDI exceeds the stronger of Direct and Code Agent by 3.8–10.9 points (Table 20). For MiniMax the stronger one is Direct, which GBDI beats by 6.7 points, so the gain does not depend on its weaker Code Agent. That Code Agent rarely fails (3 of 313 runs) but writes only about 1.6k output tokens per instance, compared with 24.6k when MiniMax answers directly. So this can explain why MiniMax's Direct outperformed its Code Agent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.