- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies error discovery and handling in LLM-based question answering over imperfect tables. Using human-reviewed RADAR-T examples, the authors adjust the positions and spacing of relevant error rows while keeping the table contents and correct answers unchanged. They find that these changes affect whether models discover all relevant errors, with more pronounced effects in the Direct mode. Another set of experiments compares original tables, tables with verified errors marked, and tables with human-reviewed repair plans applied. The accuracy gap between the latter two conditions indicates that providing error locations alone is still insufficient to ensure reliable answering. Based on these findings, the authors propose GBDI: it independently inspects five random row permutations, merges candidate error reports, and uses explicit handling guidance to direct a code agent to verify and handle relevant errors before computing the answer. The controlled experiments involve three large language models, while the full QA evaluation covers five models and all 313 RADAR-T examples. Compared with the Code Agent baseline, GBDI improves accuracy by 3.8–18.5 percentage points, with paired 95% confidence intervals excluding zero for four systems.
Critical strengths
S1. The controlled experiments directly examine error discovery and subsequent handling. The authors vary the positions and spacing of critical error rows while keeping the table contents, questions, and correct answers unchanged, testing whether error discovery is affected by row arrangement. They also compare error-marked inputs with human-repaired inputs to examine the QA difficulties that remain after error locations are provided. These controls provide concrete evidence for understanding where models still fail and inform the subsequent workflow design. S2. The practical effectiveness of the workflow is supported by evaluation across models and component controls. Across five systems and 313 RADAR-T examples, GBDI improves accuracy over the Code Agent baseline by 3.8–18.5 percentage points, with paired 95% confidence intervals excluding zero for four systems. The authors also compare random row permutations with repeated inspections under the same call budget and compare different handling guidance while keeping the error ledger fixed. These results support the practical usefulness of the workflow across multiple systems, while the component controls help examine the additional effects of row arrangement and guidance.
Critical weaknesses
W1. The methodological novelty of GBDI is limited. Its main procedure is to inspect multiple shuffled row orders, merge candidate errors, and use manually written prompts to guide a code agent in processing the table. These choices follow from the preceding experimental findings and have practical value, but the additions mainly concern workflow organization and prompt design. The paper does not sufficiently explain what GBDI advances technically over existing approaches involving repeated inspection and task-oriented data cleaning. Thus, the paper contributes more through its analysis of the problem, while the novelty of the method itself is relatively limited. W2. The reliability of the critical decisions to modify or remove records is not sufficiently examined. GBDI retains candidates proposed by any inspection, including records nominated only once. The final agent therefore needs to distinguish actual errors from false positives before deciding whether to modify, remove, or retain a record. Although the paper provides detailed rules requiring evidence verification and checks of relationships between fields, these judgments and the checking code are still produced by the same LLM. Evaluation of the handling stage mainly considers whether the final answer is correct, without further examining whether the actual repairs are correct, whether valid values are unnecessarily changed, or whether valid records are mistakenly removed. For a paper emphasizing error handling, this stage needs more direct analysis to establish that the workflow improves the reliability of data processing.
Questions for authors
Q1. How were the rules in the Intervention Skill developed? Were they fixed before the final evaluation? Were RADAR-T evaluation examples or human-reviewed repair plans used when developing or revising them? Q2. GBDI retains candidate errors nominated by only one inspection. For false-positive nominations, does the final agent make unnecessary changes to valid data or remove valid records? Q3. Table 13 shows that many questions are still answered incorrectly even after all critical rows have been discovered. Where do these failures mainly occur: incorrect data repairs, or subsequent computation or code execution errors? Q4. Beyond the number of calls, how much additional overhead does GBDI incur compared with the standard Code Agent?
6 comments
Rebuttal by Authors [2/3]
R to W2: Reliability of the decisions to modify or remove records
"Although the paper provides detailed rules requiring evidence verification and checks of relationships between fields, these judgments and the checking code are still produced by the same LLM. Evaluation of the handling stage mainly considers whether the final answer is correct, without further examining whether the actual repairs are correct, whether valid values are unnecessarily changed, or whether valid records are mistakenly removed."
Thank you for the suggestion to evaluate the handling stage directly.
TLDR. Checked against the independent human-reviewed references, the changes the agent makes are mostly correct. Edited values are rarely wrong, valid data are rarely changed, and the main failure is leaving a critical row unrepaired rather than making a harmful change. The Skill also repairs better than generic guidance.
1. Are the repairs correct? We re-executed the recorded code of every trajectory on the original table to recover the table on which the agent computed its answer (1,561 of the 1,565 trajectories are reproduced exactly), and compared it with the reviewed actions and values on the 154 reviewed instances. The results of this and the next two analyses are reported in Appendix D.7 of the revised manuscript.
| Critical rows repaired exactly, % | EDIT rows with a wrong value, % | Accuracy when all critical rows are exact, % | |
|---|---|---|---|
| Qwen Plus | 57.7 | 7.7 | 98.3 |
| GLM-4.5-Air | 46.0 | 13.0 | 82.2 |
| MiniMax-M2.5 | 21.0† | 21.9 | 65.5 |
| GPT-5 Mini | 52.0 | 8.5 | 96.6 |
| DeepSeek-V4.1-Flash | 52.2 | 7.0 | 100 |
† MiniMax often corrects values only inside its answer code without saving them to the table, so this is a lower bound.
2. Are valid values changed or valid records removed? We treat rows that RADAR did not perturb as valid and examine all 313 instances.
| Single-view false nominations changed, % | Multi-view false nominations changed, % | Error-free tables with any change | Accuracy on error-free tables | |
|---|---|---|---|---|
| Qwen Plus | 3.9 | 26.8 | 9/53 | 49/53 |
| GLM-4.5-Air | 1.8 | 20.9 | 5/53 | 48/53 |
| MiniMax-M2.5 | 4.7 | 24.4 | 0/53 | 50/53 |
| GPT-5 Mini | 11.1 | 44.4 | 5/53 | 49/53 |
| DeepSeek-V4.1-Flash | 7.7 | 42.7 | 0/53 | 52/53 |
False nominations from a single view are almost always left untouched. Rows nominated by several views are changed more often. Since unperturbed rows may still contain problems in the source data, these rates are upper bounds on unnecessary changes. On the 53 error-free tables, accuracy stays at 48–52.
3. Does the guidance itself improve the repairs? We held the candidate list fixed and replaced the Skill with generic guidance.
| Same Random-5 ledger | Rows repaired exactly, % (Generic → Skill) | EDIT rows with the exact value, % (Generic → Skill) |
|---|---|---|
| Qwen Plus | 55.9 → 57.7 | 45.6 → 56.0 |
| GLM-4.5-Air | 43.5 → 46.0 | 48.4 → 55.1 |
| MiniMax-M2.5 | 13.5 → 21.0 | 15.8 → 22.8 |
The Skill improves exact repairs for all three systems, mainly by recovering the correct value. For Qwen it also halves the changes to rows without injected errors, from 4.8% to 2.3% of such rows.
4. Verification by the same model. The reference above is independent of the model, so points 1–3 measure the quality of its self-verified decisions directly. The changes it makes are mostly correct, and only 3 of the 203 wrong answers after complete discovery follow changes to valid rows (see Q3). Its main failure is leaving critical rows unrepaired, not modifying or removing valid data. In addition, every action must cite evidence visible in the table, is recorded in an intervention log and is executed as code on a copy of the table, so each change can be audited.
R to Q1: Development of the Intervention Skill
"How were the rules in the Intervention Skill developed? Were they fixed before the final evaluation? Were RADAR-T evaluation examples or human-reviewed repair plans used when developing or revising them?"
Thank you for this question.
The rules target the general task rather than individual instances. They state when a value may be corrected (EDIT), excluded (DROP) or kept (NO-OP), add recovery rules for missing values, bad values and formatting, and contain no table, column or value-specific content (Appendix F.5). We developed them with Qwen Plus on a small subset of RADAR-T. The Skill was frozen after development, and the human-reviewed repair plans of Section 2 were completed after the freeze and were never used to develop or revise the rules.
Rebuttal by Authors [3/3]
R to Q2: Unnecessary changes from single-view nominations
"GBDI retains candidate errors nominated by only one inspection. For false-positive nominations, does the final agent make unnecessary changes to valid data or remove valid records?"
Thank you for the question.
TLDR. Rarely. The agent leaves 89–98% of single-view false nominations untouched, and accuracy on error-free tables stays at 48–52 of 53.
We treat rows that RADAR did not perturb as valid. Across the five systems, only 1.8–11.1% of single-view false nominations are changed (Appendix D.7). False nominations repeated across views are changed more often (21–44%), and since unperturbed rows may still contain problems in the source data, this rate is an upper bound. The number of supporting views is informative as well, since the share of truly erroneous rows rises from 23–60% among single-view nominations to 78–100% among rows nominated by all five views. This is why GBDI keeps single-view candidates for verification but orders the list by support.
R to Q3: Failures after complete discovery
"Table 13 shows that many questions are still answered incorrectly even after all critical rows have been discovered. Where do these failures mainly occur: incorrect data repairs, or subsequent computation or code execution errors?"
Thank you for the question.
TLDR. Mainly in the repairs. Of the 203 wrong answers after complete discovery, 92% involve a critical row not repaired as reviewed, and only 6.4% follow a fully correct repair.
We assign each wrong answer to the first stage that fails. This breakdown refines Table 13 and has been added to Appendix D.7 of the revised manuscript. This result is also coincide with our conclusion that within the repairs, choosing the action is harder than carrying it out.
| Wrong answers after complete discovery | Qwen | GLM | MiniMax | GPT-5 Mini | DeepSeek | Total |
|---|---|---|---|---|---|---|
| A critical row not repaired as reviewed | 39 | 48 | 34 | 40 | 26 | 187 (92.1%) |
| Valid rows changed | 0 | 3 | 0 | 0 | 0 | 3 (1.5%) |
| Wrong after an exact repair (computation or code error) | 0 | 3 | 8 | 2 | 0 | 13 (6.4%) |
R to Q4: Overhead beyond the number of calls
"Beyond the number of calls, how much additional overhead does GBDI incur compared with the standard Code Agent?"
Thank you for the question.
TLDR. GBDI in 5 views uses 2.6–4.0 times the Code Agent's tokens, about half on discovery. Verification adds little. However, fewer views can reduce the cost.
| Tokens per instance | Direct | Code Agent | GBDI (5 views) | GBDI's discovery part |
|---|---|---|---|---|
| Qwen Plus | 23.8k | 27.1k | 90.4k | 45.8k |
| MiniMax-M2.5 | 30.9k | 36.3k | 94.5k | 43.4k |
| GPT-5 Mini | 8.1k | 20.7k | 83.2k | 44.6k |
Each discovery view costs about 7–9k tokens. Verifying a longer candidate list adds little because all candidates are checked within one turn. For Qwen, the answering stage uses 42.2k tokens without a list and 44.6k with the five-view list. Since the gain in discovery diminishes after two or three views, using fewer views is a reasonable way to reduce the overhead. We have added the token comparison to Appendix D.8 of the revised manuscript.
We sincerely hope that these clarifications and new results address your concerns, and we would be glad to discuss any remaining questions. Thank you again for your time and valuable feedback.
Rebuttal by Authors [1/3]
We sincerely thank the reviewer for the careful summary and for recognizing that our controlled experiments directly examine error discovery and handling, and that the effectiveness of the workflow is supported across models and component controls. To address the concerns, we carried out additional analyses and updated the manuscript accordingly (changes are highlighted in blue). Each response starts with a TLDR, followed by the details.
R to W1: Methodological novelty of GBDI
Thank you for the suggestion to clarify what GBDI adds beyond existing approaches.
TLDR. Each GBDI component targets a failure mechanism found in our analysis and is tested against the corresponding existing approach. GBDI beats repeated inspection at the same budget and makes record-level decisions that column-level task-oriented cleaning cannot express.
1. The analysis behind GBDI is new. Our study asks whether an LLM finds and correctly handles the errors that affect a specific answer. Order-sensitivity studies on clean tables and answer-level benchmarks do not examine this question. With a human-reviewed reference of 154 instances, 1,162 critical rows and reviewed EDIT or DROP decisions, we identify two failure mechanisms. First, discovery depends on where the same errors appear. Back placements raise complete discovery by 11.6–16.7 points, and dispersed layouts lower it by 7.9–16.2 points. Second, handling is the larger bottleneck. Supplying repaired tables raises Code Agent accuracy to 92.9–98.1%, whereas marking the same errors changes it by at most 11.0 points.
2. What GBDI advances over repeated inspection. Repeated inspection reduces random noise but keeps every row in the same place, so an error that is hard to see at its position stays hard to see. GBDI makes three changes and tests each of them.
3. What GBDI advances over task-oriented data cleaning. Task-oriented cleaning such as AutoDCWorkflow solves a different task. It outputs a cleaned table through column-level operations, whereas GBDI answers the question and must decide for each erroneous record whether its value can be recovered from the table or the record must be excluded. In 41.7% of the 1,180 reviewed repair actions the record has to be excluded, which column-level operations cannot do. The Intervention Skill encodes these record-level decisions per error type, and its contribution is measured directly.
With the same ledger, the Skill also improves accuracy for GLM (144 to 155 correct answers) and MiniMax (140 to 160), which were not used to develop it.
We have stated these points in the new Discussion section of the revised manuscript.