acceptodds
Under review as a conference paper at ICLR 2027

CHAT: Automated, Constrained and Hard-Label Attacks on Tabular Models

Abstract

Evaluating tabular models' adversarial robustness is particularly challenging, as input perturbations must satisfy domain constraints (i.e., semantic relationships between features) to reflect realistic samples. Most existing attacks rely on manually created constraints which are inherently limited. Moreover, as they rely on model gradients or output scores, existing attacks are inadequate for practical settings where only hard labels are available. We address the first challenge by leveraging a large language model (LLM) to automatically mine rich, high-quality constraints. Our evaluation of constraints derived for three datasets from TabularBench shows that LLM-generated constraints almost fully capture manual constraints, cover additional critical constraints not found manually, and score highly on quality assessment by domain experts. We address the second challenge by proposing the first hard-label practical black-box attack on tabular models. By interweaving a misclassification-aware projection method for satisfying constraints and a leading hard-label attack from vision (CGBA), our attack, termed CHAT, achieves higher constrained attack success rates than the state-of-the-art white-box attack (CAA) on three datasets and eight models each, both defended and undefended.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
tabular models, adversarial machine learning, domain constraints

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_whjnOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses constrained adversarial evaluation of tabular models on two fronts. It uses an LLM to mine domain constraints automatically instead of relying on manually written ones, and it proposes CHAT, by interleaving a misclassification-aware projection with a hard-label base attack adapted from CGBA.

Critical strengths
  1. Automating constraint mining with an LLM is a well-motivated response to the limited coverage and precision of manually written tabular constraints.
  2. The hard-label setting is realistic for deployed tabular services and is under-explored relative to white-box tabular attacks.
Critical weaknesses
  1. Constraint mining relies entirely on the LLM's prior knowledge, and all three datasets come from domains the LLM plausibly knows well. The paper does not test a domain where the LLM would lack knowledge, so it remains unknown whether the approach degrades there.
  2. The learned constraints come from a single LLM under a single prompting strategy, and no sensitivity analysis over the model or the prompt is reported, so it is unclear how much of the reported constraint quality is attributable to the specific model and prompt used.
  3. Constraint quality is judged by one professional per domain, on 20 sampled learned constraints plus two LLM judges, with no inter-annotator agreement reported, no judging instrument provided.
Questions for authors

As above.

1 comment

W1: "... mining relies entirely on the LLM's prior knowledge, and all three datasets come from domains the LLM plausibly knows well. The paper does not test a domain where the LLM would lack knowledge, so it remains unknown whether the approach degrades there."

We concur that this may be a limitation and will state it more clearly. We chose OpenAI’s o3 model (and gave it no internet access) to prevent memorization of published benchmark constraints, as o3’s knowledge cut-off predates TabularBench’s publication date. Based on our experiments, the LLM produced high-quality constraints in three distinct domains, giving us confidence that it will succeed in others, too. If the reviewer has in mind another domain and a dataset that would be more challenging for LLMs, we will do our best to incorporate it in the revision.

W2: "... constraints come from a single LLM under a single prompting strategy… so it is unclear how much of the reported constraint quality is attributable to the specific model and prompt used"

We agree that sensitivity to the LLM, prompt, and repeated sampling is important to assess. A comprehensive study would require repeated mining runs, but due to substantial API costs and overall compute costs, we conducted only one o3 mining run. As an alternative for evaluating sensitivity, we mined LCLD constraints in three independent runs with a small model (Anthropic’s Haiku 5.5), and got roughly consistent constraints across the runs (>98% pair-wise completeness and soundness, >90% true rate by 5 LLM judges). We will acknowledge this point and include the detailed experimental results in the revision’s appendix.

W3: "... quality is judged by one professional per domain, on 20 sampled learned constraints plus two LLM judges, with no inter-annotator agreement reported, no judging instrument provided"

Table 7 (Appendix B.5) already reports the judgment categories, sample counts, and full aggregate results–we will make sure it has a clear reference in the revision. We will also include results from three additional LLM judges, reported separately from the human assessments, which gave consistent results with other judges. Moreover, we independently validated the expert feedback on the WiDS dataset with another expert and received 100% agreement; we will note this in the revision.

Reviewer_xBtpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to generate adversarial attacks for models trianed on tabular datasets that adhere to the underlying data constraints. The proposed algorithm revolves between generating an attack datapoint and then repairing it to make it respect the constraints that the data imposes (for which they use an LLM to extract the constraints and verify on the held-out data). As one can see these two constraints might be in conflict with one another (misclassification and respecting data constraints), the paper goes through a bunch of cycles to ensure both the constraints are met at the same time (they do this iteratively).

There are several axes to the evaluation that they do:

  1. Checking if the relations infereed from the data is valid and also verified by a domain expert -- I would argue this is more of a check of the reasoning capability of the underlying LLM rather than the authors contribution.
  2. success metric which is the combination of misclassification rate under adherence to constraints.
  3. Clean accuracy under the two training settings.
Critical strengths
  1. The main strength of the paper is in coming up with this algorithm that combines an LLM with gradient-based optimization to find adversarial examples focused on tabular datasets.
Critical weaknesses

I think there are several weaknesses of the paper in its current state:

  1. First of all most of what this paper describes has been covered a long time back in the counterfactual explanations community -- exactly the same problem find a misclassification that's close to the original datapoint which adheres to data constraints. See this survey paper for discussion: https://dl.acm.org/doi/full/10.1145/3677119
  2. I think this paper's techniques of using the LLM at the first stage is novel, but that's mostly an LLM capability, not a contribution of this work.
  3. Lack of performance comparison to the papers from the counterfactual explanation landscape that aim to achieve the same thing is a limitation.
Questions for authors

Please compare to methods from the counterfactual explanations landscape, there are several methods that can achieve this https://arxiv.org/abs/2106.03962.

2 comments

W1 / W3 / Q1: "most of what this paper describes has been covered a long time back in the counterfactual explanations community" and "Lack of performance comparison to the papers from the counterfactual"

Thank you for raising this point. We are aware of this body of work and apologize for the oversight. We will cover counterfactual explanations and algorithmic recourse in more depth and cite the relevant work. Generally put, works in the counterfactual area, including those in the cited survey, either focus on manually crafted constraints, ones that have limited expressiveness (e.g. denial constraints), or ones that require extensive domain knowledge/information (e.g. causal graphs). Our contribution (using highly expressive constraints that are automatically mined) could thus potentially have applications in the realm of counterfactuals as well. To our knowledge, no work in the context of either counterfactuals or adversarial examples possesses our traits of highly expressive constraints that are automatically mined, nor supports such constraints with label-only access. We will mention counterfactuals as possible future work.

We consequently do not see a direct competitor in the realm of counterfactuals to which we could compare experimentally in a meaningful way. However, if the reviewer sees fit that we compare experimentally to a specific work on counterfactuals, we would be grateful if they point it out.

W2: "... that's mostly an LLM capability, not a contribution of this work."

We concur that producing high-quality LLM-generated constraints relies on existing LLM capabilities, and we should have stated our contribution more precisely. Our work does not only make the insight that LLM-generated constraints may be more adequate than manually crafted ones for assessing the adversarial robustness of tabular models. Our contribution also lies in extending prior constraint-mining work through validation, repair, and filtering, and integrating the resulting pipeline into our attack framework.

Thank you for the constructive feedback! We have uploaded a revision with the following changes to address the weaknesses and questions the review raised (changes are colored in blue):

  • W1; W3/Q1 — Counterfactual explanations and comparisons: Section 2 now expands our discussion of counterfactual explanations and algorithmic recourse, including Verma et al. (2022) and FastAR. We also discuss Karimi et al., FACE, and Asael et al., clarifying differences in required domain knowledge and constraint expressiveness. Our response to W1/W3/Q1 further explains the challenges of direct experimental comparison. Sections 2 and 6 also identify how our constraint-mining and repair pipeline may benefit counterfactual methods.
  • W2 — Contribution beyond existing LLM capabilities: Our contribution includes constraint validation, repair, filtering, and integration into constrained attacks, as described in Section 4.1 and Appendix C. The new experiments in Table 8 (Appendix B.6) additionally quantify the contributions of saved LLM repairs and online LLM guidance to attack success and runtime.

We hope our changes address the reviewer's concerns, and welcome any further comments the reviewer may have.

Reviewer_gcfeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies two challenges in evaluating the adversarial robustness of tabular models: the limitations of manually created constraints and the inadequacy of prior attacks in settings where only hard labels are available. The authors address the first challenge by leveraging Large Language Models (LLMs) to automatically mine rich, high-quality constraints, which are further assessed and rated highly by domain experts.

The authors address the second challenge by proposing a hard-label black-box attack on tabular models. This is achieved by combining a misclassification-aware projection method, which aims to ensure that constraints are satisfied while preserving misclassification, with a hard-label attack adapted from earlier work. According to the authors, the resulting attack, called CHAT, achieves higher constrained attack success rates than the state-of-the-art white-box attack in most evaluated settings.

Critical strengths
  1. The idea is well-presented, well-motivated, and relatively easy to follow.
  2. The paper addresses the potential threats of deploying tabular models in the real world and the challenges of generating adversarial examples. The authors' insistence on ensuring that the generated adversarial examples satisfy domain-specific constraints makes them more realistic threats.
  3. The post-projection misclassification recovery and constraint-informed restart mechanisms proposed by the authors seem reasonable and potentially useful.
Critical weaknesses
  1. While the reviewer acknowledges the potential novelty of some of the components in CHAT, the paper's overall contribution appears incremental.
  2. The selection of baselines is limited. The authors mainly compare CHAT with CAA and its components. However, prior works such as [1, 2] could also serve as relevant baselines.
  3. While the authors evaluated the quality of the LLM-generated constraints using completeness, soundness, and domain-expert assessments, their experiments did not clearly establish whether the LLM-generated constraints improve adversarial attack performance.
  4. The paper introduces various components, such as cyclic projection, misclassification-aware recovery, and multi-hop boundary walking. However, the authors did not provide sufficient ablation studies to reveal the individual contributions of these components.
  5. It is unclear why the authors included initially misclassified inputs when computing the attack success rate. This could potentially overestimate the effectiveness of the attack.

References

  1. Ben-Tov, M., Deutch, D., Frost, N. and Sharif, M., 2024, May. Cafa: Cost-aware, feasible attacks with database constraints against neural tabular classifiers. In 2024 IEEE Symposium on Security and Privacy (SP) (pp. 1345-1364). IEEE.

  2. Itzhakev, Y., Giloni, A., Elovici, Y. and Shabtai, A., 2025. Addressing key challenges of adversarial attacks and defenses in the tabular domain: A methodological framework for coherence and consistency. Applied Soft Computing, p.113998.

Questions for authors
  1. Could the authors provide ablation studies on the individual components constituting CHAT?
  2. Could the authors consider comparing CHAT with the baselines listed in Weakness 2?
  3. Could the authors explain the rationale for including initially misclassified inputs when calculating the attack success rates?
  4. Can the authors experimentally verify that LLM-generated constraints actually improve adversarial attack performance? For example, the authors could compare CHAT-WB and CHAT-BB using manually defined constraints against the same attacks using LLM-generated constraints.
1 comment

W1: “... the paper's overall contribution appears incremental.”

We respectfully claim that CHAT makes at least three critical, non-trivial contributions:

(1) To our knowledge, CHAT is the first label-only attack for tabular models to attain misclassification while satisfying relational (both arithmetic and logical) constraints under a perturbation bound. Accordingly, CHAT is more adequate than prior methods for assessing model susceptibility to evasion in practical settings.

(2) CHAT mines high-quality relational constraints with LLMs and empirically validates them, significantly improving upon the coverage of manual constraints without sacrificing constraint quality.

(3) CHAT integrates LLMs into projection and offers misclassification-aware projection, allowing its white- and black-box variants to surpass the constrained success rates of state-of-the-art white-box attacks.

W2 / Q2: “... However, prior works such as [1, 2] could also serve as relevant baselines.” and “... consider comparing CHAT with the baselines listed in Weakness 2?”

We thank the reviewer for highlighting these relevant papers, including the work of Itzhakev et al., which we will add to §2 and Table 5. Our baseline selection builds on TabularBench and its evaluation setting with complex relational constraints. CaFA’s published denial-constraint predicates do not directly encode our arithmetic relations, and its DC mining can become computationally prohibitive due to exponential scaling in the number of features (see §5.1). Itzhakev et al.’s framework would require dataset-specific adaptations and relies on scores returned by the model, rendering it inadequate for label-only settings.

W3 / Q4: “... did not clearly establish whether the LLM-generated constraints improve adversarial attack performance.” and “... compare CHAT-WB and CHAT-BB using manually defined constraints against the same attacks using LLM-generated constraints.”

Thank you for the note, we will add an appendix section with a comparison of golden versus learned constraints under attack, evaluating both against the same golden constraints. Due to their higher coverage, results suggest a modest drop in constrained success rates (<4% across the three datasets) when using learned constraints rather than manual constraints. However, attacks incorporating LLM-constraints satisfy a wider range of real-world domain constraints not covered by manual constraints.

W4 / Q1: “... the authors did not provide sufficient ablation studies to reveal the individual contributions of these components” and “… provide ablation studies on the individual components?”

Thanks for raising this point. We should have described the existing ablations more clearly. Appendix B.6 includes partial ablations, which we have already extended and will include in the revision. In a nutshell, cyclic projection, misclassification-aware recovery, outer retries, multi-hop search, LLM guidance, and additional search directions all improve mean constrained success rate or runtime.

W5 / Q3: “the authors included initially misclassified inputs when computing the attack success rate” and “explain the rationale”

Retaining initially misclassified inputs that satisfy the constraints is consistent with established robustness-evaluation practices (e.g., TabularBench, AutoAttack, and RobustBench). Excluding these inputs leaves the attack rankings in Tables 3 and 4 unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.