- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses constrained adversarial evaluation of tabular models on two fronts. It uses an LLM to mine domain constraints automatically instead of relying on manually written ones, and it proposes CHAT, by interleaving a misclassification-aware projection with a hard-label base attack adapted from CGBA.
Critical strengths
- Automating constraint mining with an LLM is a well-motivated response to the limited coverage and precision of manually written tabular constraints.
- The hard-label setting is realistic for deployed tabular services and is under-explored relative to white-box tabular attacks.
Critical weaknesses
- Constraint mining relies entirely on the LLM's prior knowledge, and all three datasets come from domains the LLM plausibly knows well. The paper does not test a domain where the LLM would lack knowledge, so it remains unknown whether the approach degrades there.
- The learned constraints come from a single LLM under a single prompting strategy, and no sensitivity analysis over the model or the prompt is reported, so it is unclear how much of the reported constraint quality is attributable to the specific model and prompt used.
- Constraint quality is judged by one professional per domain, on 20 sampled learned constraints plus two LLM judges, with no inter-annotator agreement reported, no judging instrument provided.
Questions for authors
As above.
We concur that this may be a limitation and will state it more clearly. We chose OpenAI’s o3 model (and gave it no internet access) to prevent memorization of published benchmark constraints, as o3’s knowledge cut-off predates TabularBench’s publication date. Based on our experiments, the LLM produced high-quality constraints in three distinct domains, giving us confidence that it will succeed in others, too. If the reviewer has in mind another domain and a dataset that would be more challenging for LLMs, we will do our best to incorporate it in the revision.
We agree that sensitivity to the LLM, prompt, and repeated sampling is important to assess. A comprehensive study would require repeated mining runs, but due to substantial API costs and overall compute costs, we conducted only one o3 mining run. As an alternative for evaluating sensitivity, we mined LCLD constraints in three independent runs with a small model (Anthropic’s Haiku 5.5), and got roughly consistent constraints across the runs (>98% pair-wise completeness and soundness, >90% true rate by 5 LLM judges). We will acknowledge this point and include the detailed experimental results in the revision’s appendix.
Table 7 (Appendix B.5) already reports the judgment categories, sample counts, and full aggregate results–we will make sure it has a clear reference in the revision. We will also include results from three additional LLM judges, reported separately from the human assessments, which gave consistent results with other judges. Moreover, we independently validated the expert feedback on the WiDS dataset with another expert and received 100% agreement; we will note this in the revision.