- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
TabClean cleans tabular data without training a model. A profiler summarizes the table, and five role-specific agents write a Python program made of guarded clauses, each pairing a deterministic transformation with a guard that limits where it applies. Candidate programs run on annotated development rows, cell-level false positives, false negatives and regressions drive the next revision, and a rule-based decider keeps the program with the best development F1. The final program runs over the full table without further LLM calls. On six benchmarks TabClean reports a mean F1 of 0.842 and the best F1 on five, against 0.078 for Cocoon and 0.386 for AutoDCWorkflow with the same development examples.
Critical strengths
- LLM inference cost is amortized through program reuse. LLM calls sit in a bounded synthesis and debugging phase, and what remains is a program that can be rerun, about 0.48 dollars per dataset, 4.42 minutes on the 200,000-row Tax table, and 18 LLM calls on AddressTable against 5,915 for IterClean.
- Splitting a repair into a transformation and an explicit guard makes over-correction something the program can check clause by clause. The error analysis shows what unguarded edits cost Cocoon and AutoDCWorkflow under exact matching, through changes of case and date format.
Critical weaknesses
- The clause structure and evidence requirements are specified through prompts, without a formal mechanism enforcing structural compliance or guard correctness. The Coding Agent writes Python under a contract stated in its prompt, the guards and evidence levels are natural-language instructions in the skills and prompts, and the paper itself says this discipline does not formally guarantee correct guards or repairs. Apart from the LLM Review Agent, no component checks that generated code contains the guards it claims. The rest of the system, from the role split to the debugging loop and the execution-based selection, follows the agent literature the paper cites. What is new is a set of pattern-level rules for unseen values and cross-column clauses plus a loop that refines guards, and the paper never walks through a full synthesis episode, so a reader cannot tell this apart from careful prompt design.
- The standard comparison gives the systems very different amounts of supervision, so its F1 gaps do not isolate repair ability. Half of the 30 baseline entries in Table 3 round to 0.00, Cocoon scores 0.15 or lower everywhere while its own paper reports higher scores on five of the same benchmarks under a lenient protocol, and HoloClean scores 0.00 on five datasets even with oracle error locations. TabClean uses 110 to 369 annotated development rows, against 20 tuples for Baran and 5 for IterClean. Table 4 partially addresses this concern by giving Cocoon and AutoDCWorkflow the same development examples and test rows. However, comparable supervision-matched results are still missing for Baran and IterClean.
Questions for authors
- Why do half of the baseline entries round to 0.00, including Cocoon on benchmarks where its own paper reports higher scores and HoloClean with oracle error locations? How do the comparisons change under a matching rule that tolerates case and null-token differences?
- How do Baran and IterClean perform when given the same development rows as TabClean?
- What do the guards contribute on their own? For example, how do precision and F1 change when the guard requirements and evidence levels are removed from the prompts, and how often does each guard family appear in the generated programs?
- Is there evidence that a committed program stays accurate on a later batch, or on a batch drawn differently, without being synthesized again?
The authors have not yet responded to my comments, so I have decided to maintain my original score.