acceptodds
Withdrawn from ICLR 2027

Efficient and Interpretable Transformer for Counterfactual Fairness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The growing reliance on machine learning for decision-making in highly regulated industries has raised fundamental challenges in regulatory fairness and interpretability compliance. Although attention-based transformers provide a powerful framework for modeling complex feature dependencies, existing debiasing methods for tabular data either fail to guarantee counterfactual fairness or rely on restrictive causal assumptions that limit practical applicability. We propose Feature Correlation Transformer (FCorrTransformer), an efficient and attention-light architecture specifically designed for tabular learning, where the attention matrix offers direct interpretation as pairwise feature dependencies. Building on this structure, we introduce Counterfactual Attention Regularization (CAR), a regularization framework that enforces group-invariant representations of sensitive features at the attention level, thereby yielding counterfactually fair predictions. We further develop an optimized input augmentation strategy for efficient counterfactual attention learning with minimum computation resource. Empirical experiments on both imbalanced classification and regression benchmarks demonstrate that FCorrTransformer+CAR achieves strong counterfactual fairness guarantees with preserved predictive performance, offering a practical framework for responsible AI in regulatory-sensitive domains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
algorithm fairness, insurance, tabular learning, transformer

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_mKAuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed a new model for tabular fairness. They designed it as a tabular transformer in which each feature gets a single scalar embedding. Attention is computed directly from these scalars with no query, key or value projections, using a single head and no residual connection around attention. The authors argue this makes the attention matrix readable as pairwise feature dependencies. On top of this they propose CAR, a regularizer that penalizes changes in the sensitive feature’s attention column when the sensitive attribute is switched to other categories, plus a cheaper single-pass version for training. A theorem states that a small CAR loss bounds the change in prediction when only the sensitive attribute changes. Experiments on BAF, an insurance dataset and Adult show near-zero counterfactual gaps for FCorrTransformer+CAR with a small drop in AUROC and AUPRC compared to FT-Transformer.

Critical strengths
  • The most valuable point of this paper is the evidence showing regularizing attention in a standard transformer does not make predictions invariant, because information about the sensitive attribute bypasses attention through residual paths and projections. The residual-connection ablation (Tables 7 and 8) is a useful, honest observation. FT-Transformer+CAR retains a clear gap, and reintroducing the residual in FCorrTransformer brings the gap back. This is a fair caution for the attention-based debiasing literature. Although by itself it is a modest contribution.
Critical weaknesses
  • There are issues in the evaluation pipeline that hurt paper claims. The paper cites Kusner et al. for counterfactual fairness, but the definition in Section 2 and the metrics in Section 4 only switch the sensitive column while keeping every other feature fixed. The authors themselves call the result “direct counterfactual fairness” after Theorem 1. This only checks whether the model reads the sensitive column, and any model that ignores the column satisfies it perfectly. Kusner’s definition relies on a causal model where features influenced by the attribute also change, and that indirect path is exactly what matters in practice. The paper’s own synthetic setup is built around indirect discrimination (X1 affects y only through X2), yet the metric cannot measure it. The same applies to the insurance motivation, since the regulations cited in the introduction are mainly about proxy discrimination. Simply dropping the sensitive feature is therefore the natural baseline. The paper reports it in the appendix (Table 8, “+ Rem”), and it also gives zero gaps with essentially the same AUROC and AUPRC as CAR. Group fairness does not improve much either: in Table 3, test DPD and EqOdd for FCorrTransformer+CAR remain high (0.106 and 0.659), and the plain FFN baseline is actually lower on both. I do not see what CAR offers beyond removing the column.

  • There are some real concerns about the interpretability claim and the baseline comparison. Because each embedding is a single scalar, the attention score between two features is just the product of their two scalars. The heatmaps therefore show how the learned scalars co-vary, not how the features depend on each other. Figure 2 illustrates this. X1 and X3 are independent by construction, yet their entry (-0.98) is larger in magnitude than the X1-X2 entry (0.37) that the text describes as the strongest dependency. X2 and X3, which together determine y, have the largest off-diagonal entry, and the text calls these negative values negligible. A partial overlap with LightGBM feature importance is weak support. On the experimental side, the only comparisons are generic tabular models with and without CAR. No established fairness method is included, such as adversarial debiasing, the reductions approach, FairPFN or causal counterfactual-fairness methods, so it is hard to place the result.

Questions for authors
  • What does CAR achieve that dropping the sensitive feature does not? Table 8 suggests the two are equivalent on every metric reported. It would help to show the learned embedding of the sensitive feature across categories after CAR training.
  • Can you evaluate under a causal counterfactual, where features influenced by the sensitive attribute also change? The synthetic dataset is a natural place for this, since X2 depends on X1.
  • How should the Figure 2 heatmap be read as feature dependencies, given that the independent pair X1 and X3 receives a large entry?
  • Can you add at least one established fairness baseline and LightGBM to Tables 2 and 3, and state which data were used for hyperparameter tuning and threshold selection?
Reviewer_AciaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies fair and interpretable tabular models for regulated fields such as insurance. To this end, the authors propose FCorrTransformer, which uses one-dimensional feature embeddings, removes the query, key, and value projections, and removes the residual connection around attention. In addition, they propose CAR, a penalty that makes the sensitive feature column in the first attention layer the same across all sensitive categories. Theorem 1 shows that a small CAR loss bounds the change in prediction when only the sensitive value changes. Experiments were conducted on three datasets.

Critical strengths
  1. The proposed architecture is simple and lightweight.

  2. The paper explains why fair attention does not imply fair predictions in standard transformers, and the ablation results in Tables 7 and 8 supports this.

  3. The input augmentation trick is interesting. It computes all counterfactual attention columns in one forward pass, and largely cuts the regression training time per epoch.

Critical weaknesses
  1. The fairness notion used in this paper is not the counterfactual fairness in (Kusner et al.). In (Kusner et al.), counterfactual fairness is defined with a causal model where changing the sensitive attribute also changes its descendants. However, in this paper, only the sensitive input is swapped and all other features stay fixed, so only the direct use of the sensitive feature is tested. The baseline results in the paper also shows the problem: removing the sensitive feature (FCorrTransformer + Rem) gives zero gaps in Table 8, with similar test AUROC compared with FCorrTransformer + CAR. What does CAR do compared with simply dropping the sensitive feature?

  2. The proof of Theorem 1 needs further clarification. Assumption 1 bounds the change in the loss, but the proof uses it to bound the change in the prediction ff. Could you clarify how this is obtained?

  3. No standard fair learning methods are included in the comparison, such as adversarial debiasing or learning with fairness constraints. The comparisons are only architectures with or without CAR, so it is difficult to judge the fairness and accuracy trade-off.

  4. The interpretability evidence is weak. Each pre-SoftMax diagonal entry is a squared scalar embedding, so calling it "feature significance" is not well justified. Could you test this by masking each feature and measuring how much the predictions change?

[1] Kusner, Matt J., et al. "Counterfactual fairness." Advances in neural information processing systems 30 (2017).

Questions for authors
  1. Could you clarify how your definition relates to the counterfactual fairness definition in (Kusner et al.)? What does CAR offer over simply removing the sensitive feature?

  2. After CAR training, does the sensitive feature's embedding become the same for all categories? If so, CAR is close to dropping the feature.

  3. Could you fix the proof of Theorem 1 in W2?

  4. Could you add one or two standard fair learning baselines for comparison? Also, could you validate the interpretability claim as detailed in W4?

Reviewer_i3SZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FCorrTransformer, a transformer architecture for tabular data with custom attention with improved interpretability, and CAR, a regularization framework that enforces counterfactual fairness. Both are evaluated on synthetic data and on 3 case studies, 2 of which use real data.

Critical strengths

The paper's topic is timely and well-motivated by high-risk AI applications.

Critical weaknesses
  • The formulation of counterfactual fairness (lines 114-116) seems incorrect. As written, it assumes there is no causal path from the sensitive attribute to any other attribute, because a counterfactual change in a sensitive attribute would causally influence some other attribute in practice (which would also change, unlike the formalization stating x_r remains the same). For example, in the experiment using age as sensitive attribute, age likely has a causal path to other attributes like employment status. This notion of fairness could be achieved trivially by ignoring X_s in the model computation, so it is closer to "fairness by unawareness". This would be aligned with the regularization toward a group-invariant representation, which precisely makes further computation independent on the attribute's original value - makes the model unaware. This difference is critical, since fairness by unawareness is often criticized for being insufficient, including by Kusner et al. (2017).
  • Some claims are weakly supported. The method does not outperform the baseline (Table 2) which weakens the claim of method's effectiveness on a variety of data. The interpretability claim is also weak, based on individual opinion (no independent human evaluation, no comparison to ground truth or expert opinion). The paper also states that the Fig 4 feature importances largely agree with LightGBM in Fig 7, but among the top 5 only one feature is shared (current_address_months_count, ranked 1st in one and 5th in the other).
  • The baselines are incomplete: there is no non-neural baseline despite tree-based methods showing good performance. There is also no baseline aimed specifically at fairness. LightGBM was evaluated and reportedly performs better, but its results are shown only through the feature importances.
Questions for authors

Clarification requests

  • What value of the fairness bound (epsilon) was achieved by CAR, and why is it not reported?
  • Why is test-set group fairness much better for FFN than for the proposed method in Table 3?
  • Is the specialized attention used only in the first transformer block, with standard attention in the remaining blocks?
  • Why was the defined fairness measure itself not used to evaluate the experiments?
  • Why was this synthetic example used? The bias values already seem small before regularization in Table 1. What are the standard deviations there?
  • Why weren't any non-neural baseline results presented in the main body? Was LightGBM run only for feature importance comparison?
  • The paper says the approach can be extended to multiple sensitive features. Does this not cause an exponential explosion in the number of feature combinations, with the computational complexity compounded by the specialized attention?
  • Why was CAR used on FT-Transform? Does it not depend on the modified transformer architecture?

Changes that would raise my score

  • Report the achieved fairness bound (epsilon) and show that CAR actually enforces counterfactual fairness. Explain why FFN outperforms your approach on group fairness metrics.
  • Add fairness-oriented baselines and non-neural baselines (e.g. the LightGBM with full results).
  • Correct or justify the counterfactual fairness formulation (lines 114-116) relative to fairness by unawareness.
  • Correct the claimed agreement between Fig 4 and Fig 7, or support it with evidence.

Minor issues and typos

  • line 79 - a instead of an
  • writing A^1() instead of A()^1 would seem more natural
  • code should be part of the submission
  • Why were only the synthetic-data experiments kept in the main body?

Suggestions for future work (these do not affect my assessment)

  • Evaluate whether the approach can be used with tabular foundation models such as TabPFN.
  • Include a tabular foundational model as a useful, high-performing baseline.
Reviewer_C1rzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an interesting and computationally efficient approach to fairness-aware tabular learning. I appreciate the authors' efforts to integrate architectural design with fairness regularization, rather than simply imposing additional constraints on an existing model. The proposed architecture is lightweight and parameter-efficient, and the optimized input augmentation strategy demonstrates practical consideration of computational efficiency. The experimental results also show promising improvements in several fairness metrics. However, I have several concerns about the theoretical foundations and the effectiveness of the proposed method. First, the paper's definition of counterfactual fairness appears to focus primarily on prediction invariance under sensitive-attribute substitution, which may not fully align with the causal counterfactual fairness framework of Kusner's paper. Second, the practical advantages of CAR over simply removing sensitive features remain unclear, particularly given the comparable results reported for FCorrTransformer + Rem. Furthermore, the theoretical justification in Theorem 1 requires additional clarification, and the simplified attention architecture may limit the model's ability to capture complex feature interactions. Overall, while the paper explores a potentially valuable direction, I am not fully convinced that the current theoretical analysis and experimental evidence sufficiently support its main claims. A more rigorous theoretical justification, clearer discussion of the fairness definition, and more comprehensive comparisons with simpler baselines and established fairness-aware methods would significantly strengthen the paper.

Critical strengths

The paper introduces an interesting approach that translates counterfactual fairness objectives into attention-level regularization, providing an intuitive mechanism for mitigating the influence of sensitive attributes on model predictions. The proposed FCorrTransformer features a streamlined architecture with substantially fewer parameters and lower memory requirements than conventional tabular Transformers, making it computationally efficient and potentially suitable for resource-constrained applications. The paper introduces an optimized input augmentation strategy to reduce the computational overhead of counterfactual attention regularization.

Critical weaknesses

The proposed method appears to be a form of feature masking.

The proposed architecture relies heavily on one-dimensional feature embeddings and projection-free attention, which may lead to excessive simplification of the Transformer architecture. I am concerned that these design choices could undermine the core strengths of Transformers, particularly their capacity to learn expressive feature representations and model complex feature interactions.

I find the theoretical analysis somewhat unclear, particularly the proof of Theorem 1.

Questions for authors

Since Softmax is applied over different feature spaces during training and inference, I wonder whether the resulting attention distributions remain consistent. Could the authors explain why minimizing the CAR loss in the augmented feature space still guarantees counterfactual fairness during inference, as claimed in Theorem 1?

The proposed method seems similar to simply removing sensitive features. Tables 7 and 8 show that FCorrTransformer + Rem achieves zero fairness gaps with comparable predictive performance. Could the authors provide a more detailed explanation of the advantages of CAR over simply removing sensitive features?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.