acceptodds
Under review as a conference paper at ICLR 2027

Certified Feature Unlearning under Selective Feature Revocation

Abstract

Feature unlearning seeks to remove a model's predictive dependence on specified input features while preserving the utility of the remaining features. Existing feature-unlearning methods typically re-access the original values of the very feature being removed, which is often impermissible once that feature has been revoked for legal, privacy, or governance reasons. We study this selective feature revocation setting, in which the retained features and labels remain available but the per-example values of the revoked feature do not, and propose Bounded Mechanism-Covered Replacement (BMCR). BMCR replaces the revoked feature with values drawn from a declared bounded range and fine-tunes the model so that its predictions stay stable across these replacements; the declared range is the only information BMCR uses about the revoked feature, and its coverage defines the scope of the guarantee. We prove that the BMCR objective penalizes residual direct dependence on the revoked coordinate and that its near-minimizers are prediction-level close to retraining without that feature. We further show that, at any finite training checkpoint, the effect of off-manifold replacements on in-distribution risk and the model-assisted attribute-inference advantage over the retained features alone are both controlled by the same residual dependence. Experiments on seven tabular datasets, three architectures, and single-, multi-, and sequential-deletion settings show that BMCR matches retraining in utility, outperforms replacement and structural-masking baselines, and is about faster than retraining in wall-clock time.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Feature unlearning, Certified unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QqiVOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies feature unlearning when retained features and labels remain accessible but revoked feature values do not. BMCR fine-tunes a model after replacing the revoked coordinate with independent draws from a declared bounded range. The analysis relates replacement risk to direct coordinate dependence and, under additional assumptions, to prediction-level proximity to retraining and finite-checkpoint risk transfer. Experiments cover seven tabular datasets, three architectures, multiple deletion settings, structural baselines, and attribute-inference audits.

Critical strengths
  1. The access restriction is concrete and useful. The paper distinguishes declared support, existing preprocessing information, aggregate statistics, and per-example revoked values, rather than comparing methods as though they had identical access (Tables 1 and 5). The replacement rule is simple to implement under this restriction.

  2. The squared-loss decomposition gives a clear explanation of why independent replacement penalizes direct coordinate dependence. The paper also distinguishes this dependence from information recoverable through retained features and states the support and regularity assumptions. These distinctions are important for interpreting feature-unlearning results.

Critical weaknesses
  1. The structural-baseline results are incompatible with the stated baseline and metric definitions. Section 5, pp. 7–8, and Appendix W define FPE by removing the target input entirely and fine-tuning the reduced network. RIPV holds retained inputs fixed and varies only the revoked coordinate. FPE must therefore have zero RIPV under deterministic evaluation, irrespective of its utility or fine-tuning quality. Yet Table 3(a) reports 0.042, and Tables 42–48 report 0.039–0.045. The stated zero-and-freeze SFI construction presents a similar concern. The downstream-refitting explanation on p. 9 does not resolve this: even an unretrained FPE is invariant to an absent input. If stochastic evaluation causes the positive variance, it must be separated from coordinate dependence. Until the implementation, metric and tables are reconciled, this central comparison cannot establish BMCR’s advantage in direct-dependence reduction over structural removal.

  2. Table 35 (Appendix T, p. 60) needs a precise definition of its uncertainty. It compares the same removed-feature configurations at epochs 1 and 1500 and reports a paired change. If all three columns give standard deviations over the same paired observations, the change SD cannot be smaller than the absolute difference between the two epoch SDs. For RIPV, the reported SDs of 0.0112 and 0.0034 require a change SD of at least 0.0078, but the table gives 0.0060. For TRI, the corresponding lower bound is 0.034, versus the reported 0.029. Rounding cannot explain either gap. Different aggregation levels or a mixture of SD and SE could explain the numbers, but would need to be stated explicitly.

  3. Proposition 4.6(iii) has a valid response-stability argument, but its final comparison does not establish a nontrivial model-specific attribute-inference guarantee. For fixed f and queries determined by retained features, the assisted attack is itself a function of those features. Its risk is therefore already bounded below by the unrestricted retained-features-only infimum in Appendix E, p. 29, without any small-dependence assumption. The proof’s intermediate comparison with responses from the averaged predictor remains useful. However, that comparison is not the same as a feasible attacker without model access. Table 3(b) compares particular fitted attackers, not the unrestricted infimum. The theory and empirical benchmark need to be aligned; this is an interpretation issue, not a demonstrated false theorem.

Questions for authors
  1. What computation produces nonzero RIPV for FPE? Please provide the input wrapper and audit procedure, verify identical outputs when only the removed coordinate changes with model randomness fixed, and reconcile Tables 3 and 42–48. If another quantity was measured, please state it and reassess the comparison under the defined RIPV.

  2. What observation unit, sample count, aggregation rule, and uncertainty measure were used for each column of Table 35? Please reconcile the RIPV and TRI paired-change values with the two epoch columns, using the underlying paired records or a reproducible aggregation script.

  3. Is Proposition 4.6(iii) intended as stability relative to the averaged predictor, or as an advantage over a restricted attacker without model access? The former follows from the proof. For the latter, what restriction prevents the assisted predictor from belonging to the comparison class, and how does that restriction match Table 3(b)?

3 comments

The author sincerely thank the reviewer for their exceptionally careful and detailed review. Their comments led to several new analyses, which we describe below

W1 and Q1: Non-zero RIPV for SFI and FPE.

We thank the reviewer for this precise diagnosis. SFI and FPE cannot respond to xix_i, so under the defined RIPV both must be exactly zero. There is no other input path, and all methods use the same RIPV code.

What produced the non-zero values. RIPV was computed with the network in training mode, so dropout was active and every forward pass drew a fresh mask ξjr\xi_{jr}. The reported quantity was therefore 1n∑jVarr[fξjr(x−i(j),Tjr)]\frac1n\sum_j \mathrm{Var}_r\big[f_{\xi_{jr}}(x_{-i}^{(j)},T_{jr})\big], which adds dropout-induced variance to the coordinate dependence. This term is present for any network with dropout, including FPE.

Input wrapper and audit procedure (corrected).

class FPEWrapper(nn.Module):          # full input -> reduced network
    def forward(self, x):             # drop coordinate i, then call the reduced net
        return self.net(torch.cat([x[:, :self.i], x[:, self.i+1:]], dim=1))
# SFI needs no wrapper: column i of the first projection (FT-Transformer:
# the tokenizer weights of x_i) is zero and frozen during fine-tuning.

@torch.no_grad()
def ripv(model, X_audit, i, sample_q, m):
    model.eval()                      # the fix: previously missing
    v = []
    for x in X_audit:                 # retained features held fixed
        xs = x.repeat(m, 1)
        xs[:, i] = sample_q(m)        # only coordinate i changes
        v.append(model(xs).var(unbiased=True))  # bounded effective logit for classification
    return torch.stack(v).mean()

Verification. With dropout disabled, changing only xix_i leaves the output of every SFI and FPE model identical (maximum absolute difference 0 over all audit points and draws, all datasets, backbones, and seeds), so their RIPV is exactly 0.

Reconciling Tables 3 and 42–48. We recomputed RIPV in evaluation mode for all methods (TRI and MIA were re-evaluated in the same pass). In the per-dataset tables (now Tables 43–49), SFI and FPE have RIPV 0.000 on every dataset, and BMCR's RIPV also decreases because the same noise had inflated it. Table 3 is now explicitly the mean ± std across the seven datasets of these per-dataset values, so it can be reproduced from them.

W2 and Q2: Uncertainty in Table 35 (now Table 36).

For paired values over the same units, the SD of the change is at least the absolute difference of the two SDs. Equivalently, the reported values imply a correlation between epochs above 1 (1.33 for RIPV, 1.21 for TRI). The three columns therefore could not share an aggregation level, and the table did not say so.

What the original columns were. All three columns use the same n=30n=30 Magic configurations (2 removed features × 3 backbones × 5 values of ε\varepsilon, each averaged over 3 seeds).

  • Epoch 1 and epoch 1500 columns: mean ± SD over the 30 configurations.
  • Paired-change column: the mean was taken over the 30 configurations, but the ± was the mean of the five per-ε\varepsilon SDs of the per-configuration change.

Grouping by ε\varepsilon keeps only the variation within groups, so it removed the variation between values of ε\varepsilon from the change column only. That is why its SD fell below the bound. Mixing these levels in one table was our error.

Corrected table. All columns are now mean ± SD across the same 30 configurations, with the change computed per configuration (epoch 1500 − epoch 1). The means are unchanged; only the SDs of the change differ:

MetricEpoch 1Epoch 1500Paired change
RIPV0.0251 ± 0.01120.0075 ± 0.0034−0.0176 ± 0.0086
TRI0.934 ± 0.0490.990 ± 0.015+0.056 ± 0.037
EI2205.6 ± 471.22.69 ± 0.29−2202.9 ± 471.0
RASI0.754 ± 0.0860.946 ± 0.028+0.192 ± 0.079
SRI0.139 ± 0.0480.121 ± 0.040−0.018 ± 0.029
SDI0.029 ± 0.0100.018 ± 0.007−0.011 ± 0.008

Every row now satisfies the bound (e.g., RIPV: 0.0086 ≥ 0.0078; TRI: 0.037 ≥ 0.034). The caption and the text of Appendix T now state the unit, n=30n=30, the aggregation rule, and that all entries are SDs. The aggregation script and the per-configuration records are included in the supplementary material.

W3 and Q3: Interpretation of Proposition 4.6(iii) and its relationship to the empirical attack benchmark.

We agree with this precise distinction between the validity of the inequality and the informativeness of its comparator. In the fixed-model setting, the final comparison with the unrestricted retained-features-only infimum does not establish a nontrivial dependence-sensitive attribute-inference guarantee. The intermediate response-stability result remains valid and useful. We will revise the proposition and the associated empirical interpretation to make this distinction explicit.

1. The unrestricted comparison does not require small residual dependence.

Let Z=X−iZ=X_{-i} and write the model-response vector as

rf(Z)=(f(Z,t1(Z)),…,f(Z,tk(Z))).r_f(Z) = \big(f(Z, t_1(Z)), \ldots, f(Z, t_k(Z))\big).

For fixed ff and queries determined by ZZ, any assisted attack

hA,f(Z)=A(Z,rf(Z))h_{A,f}(Z)=A(Z,r_f(Z))

is itself a measurable function of ZZ. Therefore,

\ell_{\mathrm{att}}(A(Z,r_f(Z)),X_i) \right] \ge \inf_h \mathbb E\!\left[ \ell_{\mathrm{att}}(h(Z),X_i) \right]$$ already holds without a dependence-dependent remainder. Thus, although the final inequality in Proposition 4.6(iii) is mathematically valid, interpreting its remainder as a nontrivial bound on excess model-assisted leakage over this unrestricted optimum is too strong. We will remove that interpretation. **2. The substantive result is stability under replacement-averaged responses.** The intermediate argument in Appendix E supports a precise alternative statement. Define $$\bar f(z)=\mathbb E_{T\sim q}[f(z,T)], \qquad \bar r_f(z)=\bar f(z)\mathbf 1_k.$$ For the same fixed attack map $A$, consider $$\mathcal{R}_f(A) = \mathbb{E}\big[\ell_{\mathrm{att}}\big(A(Z, r_f(Z)), X_i\big)\big]$$ and $$\mathcal{R}_{\mathrm{avg}}(A) = \mathbb{E}\big[\ell_{\mathrm{att}}\big(A(Z, \bar{r}_f(Z)), X_i\big)\big].$$ Under the existing assumptions that the queries lie in $[a,b]$, $A$ is uniformly $L_A$-Lipschitz in the response vector under the $\ell_\infty$ norm, and the attack loss is uniformly $\lambda$-Lipschitz in the estimated attribute, the proof gives $$\left| \mathcal R_f(A)-\mathcal R_{\mathrm{avg}}(A) \right| \le \lambda L_A\kappa(f), \qquad \kappa(f)=2\left[L_f(b-a)D_q(f)\right]^{1/3}.$$ Indeed, $$\big\lvert\mathcal{R}_f(A) - \mathcal{R}_{\mathrm{avg}}(A)\big\rvert \le \lambda L_A\, \mathbb{E}_Z \lVert r_f(Z) - \bar{r}_f(Z) \rVert_\infty$$ $$\le \lambda L_A\, \mathbb{E}_Z \sup_{t\in[a,b]} \big\lvert f(Z,t) - \bar{f}(Z)\big\rvert$$ $$\le \lambda L_A\, \kappa(f).$$ where the final step follows from Lemma D.8 and the existing argument in Appendix E. This bounds how much removing the coordinate-dependent variation in the responses can change the risk of the same stable attack. We will restate Proposition 4.6(iii) in this form. Crucially, $\bar r_f(Z)$ is a model-dependent reference. Although it is mathematically a function of retained features, a practical attacker without model access need not know or be able to compute it. We will therefore not describe this reference as a feasible attacker without model access. Similarly, $D_q(f)=0$ implies equality between the actual and averaged response channels, but does not imply that a finitely trained attacker gains nothing from the common response $\bar f(Z)$. That response may still provide a useful learned transformation of the retained features.

3. Table 3(b) evaluates fitted attackers, not the unrestricted population optimum.

The retained-feature random forest in Table 3(b) achieves R2=0.669R^2=0.669. This is an empirical baseline produced by a particular fitted predictor, rather than the optimum over all measurable functions of ZZ. The reported results are:

Attacker inputsReported R2R^2Increase over the fitted retained-feature baseline
Retained features only0.669—
Retained features + original-model responses0.813+0.144
Retained features + retrained-model responses0.678+0.009
Retained features + BMCR-model responses0.686+0.017

These measurements support a specific empirical conclusion: under the evaluated attack protocol, BMCR substantially reduces the reconstruction improvement observed with the original model's responses, and its attack score is numerically close to that of the retrained reference. They do not estimate advantage over the unrestricted retained-feature optimum or establish statistical equivalence to retraining.

The positive increment for the retrained model is also compatible with this interpretation. Its responses are functions of retained features alone, but supplying such transformations can help a finite fitted learner. An observed improvement over a particular fitted baseline therefore does not establish additional information beyond the retained features.

We will replace “data-only floor” with “fitted retained-feature baseline” and describe “Gain” explicitly as the observed increase in attack R2R^2 over that baseline. We will also remove the statements that these increments are predicted by Proposition 4.6(iii) or demonstrate removal of almost all model-specific leakage.

The empirical attack benchmark will be distinguished from a quantitative instantiation of the stability theorem. Such an instantiation would require verification of the response-Lipschitz and loss assumptions and a comparison of the same fixed attacker under actual and averaged responses. Table 3(b) does not currently perform that comparison.

4. Scope of the revision.

We will align Proposition 4.6(iii), its proof in Appendix E, Remark 4.7, and the corresponding abstract and contribution statements around the response-stability guarantee above. Table 3(b), its caption, and Appendix X will describe comparisons among fitted attackers without identifying the empirical baseline with an unrestricted inference optimum.

These revisions concern the formulation and interpretation of the attribute-inference result. The residual-dependence decomposition, prediction-level comparison with retraining, and the realistic-input and risk-transfer results in Proposition 4.6(i)–(ii) are unaffected.

Reviewer_9XYiOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the feature forgetting problem where the original values ​​of a feature cannot be accessed after it has been revoked, and proposes the BMCR method. This method randomly replaces the target feature from a pre-declared range of values ​​and fine-tunes it using retained features and labels, gradually making the model's predictions independent of the deleted features. The authors theoretically analyze the relationship between replacement loss and residual feature dependency, and demonstrate, under certain conditions, the closeness of this method to a retrained model at the prediction level. Experimental results show that BMCR can reduce computational overhead while maintaining near-retraining performance, and outperforms some baseline methods on multiple forgetting and privacy evaluation metrics.

Critical strengths
  1. This paper considers scenarios where the original values ​​of revoked features are no longer accessible, which better aligns with practical data privacy requirements than traditional feature forgetting scenarios. The authors not only propose a simple random replacement method but also establish a theoretical link between residual feature dependencies and near-retraining at the prediction level, providing a strong theoretical basis for the method.

  2. The authors conducted experiments on seven datasets and three model architectures, covering single-feature, multi-feature, and sequential forgetting models, and organized baseline comparisons according to different levels of data accessibility. Furthermore, the paper evaluates feature relevance, attribute inference risk, and computational efficiency.

Critical weaknesses
  1. The core contribution of this paper is certified feature unlearning, but the main theoretical results rely on near-optimal solutions, replacement interval coverage, and Lipschitz conditions. Actual experiments primarily measure residual feature dependence using RIPV, without providing a complete numerical certification bound for the deployed model. Furthermore, the finite replacement intervals used in the main experiments (e.g., [−0.1,0.1][-0.1,0.1]) do not necessarily cover the actual value range of the target feature, thus only providing local guarantees.

  2. The paper emphasizes an approximately 40x speedup, but this result mainly comes from a specific dataset and early training checkpoints. Table 2 shows that as training increases from 100 epochs to 1500 epochs, RIPV further decreases, but the average efficiency index (EI) drops from 40.3 to 2.7. Considering the paper's emphasis on both forgetting guarantees and computational efficiency, the authors need to more explicitly compare the computational costs required to achieve the same level of residual dependence.

Questions for authors
  1. In Table 3(a), the RIPVs for SFI and FPE are 0.033 and 0.042, respectively. However, according to Section 5 and Appendix W, SFI has reset and frozen the input weights corresponding to the target feature, while FPE directly removes the input path of that feature. According to the definition of RIPV, the predictions of these two methods should no longer change with the replacement value of the target feature, and theoretically, the RIPV should be close to zero. Can the authors explain why non-zero results are still obtained? Are there other input paths, or does the baseline use a different RIPV calculation method?

  2. The conclusions of Theorem 4.2 and Theorem 4.3 rely on approximate optimality of the replacement risk, while the neural networks used in practice are trained through non-convex optimization. Although the paper discusses the case of finite checkpoints in Appendix E.3, the experiments mainly report empirical estimates of RIPV. Can the authors explain how to obtain numerical certification bounds with a clear confidence level for models deployed in practice? Specifically, how are optimization error, finite sample error, and the Lipschitz constant determined?

  3. Table 2 primarily uses the standardized [−0.1,0.1][-0.1, 0.1] interval, but this interval typically fails to cover all true values ​​of the target feature. Could the authors report the proportion of samples in each dataset that actually fall within the replacement interval, and the model's sensitivity to the target feature when the range exceeds the interval? Furthermore, when using BMCR-full or BMCR-MM, which cover the full range of values, can similar forgetting effects and computational efficiency still be achieved?

6 comments

The author sincerely thank the reviewer for their exceptionally careful and detailed review. Their comments led to several new analyses, which we describe below.

W1: Scope of certification, numerical validation, and replacement coverage.

We thank the reviewer for raising the distinction between theoretical guarantees and their numerical instantiation. Our guarantees are explicitly scoped by the replacement distribution, and the three conditions mentioned in the review enter different parts of the analysis. We clarify these distinctions below, together with what the reported experiments establish.

1. Near-optimality, coverage, and Lipschitz continuity have different roles.

The basic residual-dependence quantity is Dq(f)=EZ VarT∼q[f(Z,T)],Z=X−i.D_q(f) = \mathbb{E}_{Z}\,\mathrm{Var}_{T\sim q}\big[f(Z,T)\big], \qquad Z = X_{-i}.

Under squared loss, Lemma 4.1 establishes

Rq(f)=E[(fˉ(Z)−Y)2]+Dq(f),fˉ(Z)=ET∼q[f(Z,T)].R_q(f) = \mathbb{E}\big[(\bar{f}(Z) - Y)^2\big] + D_q(f), \qquad \bar{f}(Z) = \mathbb{E}_{T\sim q}\big[f(Z,T)\big].

This identity holds for any square-integrable predictor and requires neither near-optimality nor Lipschitz continuity. Theorem 4.5 and Appendix E.3 provide the corresponding arbitrary-checkpoint dependence bound for bounded effective logits.

The additional conditions serve specific purposes:

  • Near-optimality allows excess risk to control residual dependence and supports the separate prediction-distance comparison with retraining in Theorem 4.3. It is not required to directly estimate or statistically bound an achieved predictor's residual dependence.
  • Support coverage determines the feature values to which a guarantee applies. It is required when extending the interpretation from a declared replacement interval to the entire operational feature domain.
  • Lipschitz regularity is used to strengthen average dependence control into range-wide approximate guarantees and to transfer it to realistic inputs, as in Proposition 4.6 and Lemma D.8. It is not required for the average-dependence identity or a statistical audit under the specified replacement distribution.

Thus, these conditions should not be interpreted as a single set of assumptions underlying every result. We will make their distinct roles more prominent.

2. RIPV directly estimates the theoretical quantity, but a point estimate is not a complete numerical certificate.

RIPV is the Monte Carlo estimate of the same Dq(f)D_q(f) appearing in the theoretical results. It therefore directly measures the quantity controlled by our analysis. However, the reported RIPV values and their standard deviations do not, by themselves, constitute confidence-qualified numerical certificates of full-domain invariance or equivalence to retraining. Remark 4.7 and Appendix F also explicitly state that the Lipschitz constants used for range-wide conclusions are not formally certified in the experiments.

The finite-sample interpretation can be made concrete without assuming Bayes-optimal fine-tuning. Appendix E already notes that unbiased RIPV can be supplemented with a concentration term. Specifically, consider a fixed trained predictor whose outputs lie in a known interval of width BB. On nn fresh independent retained-feature audit examples, draw m≥2m\ge2 independent replacements per example and let D^q\widehat D_q be the average within-example sample variance, using the unbiased denominator m−1m-1. Each such variance is bounded by

Cm=mB24(m−1).C_m=\frac{mB^2}{4(m-1)}.

Consequently, with probability at least 1−α1-\alpha over the audit sampling,

\le U_\alpha := \widehat D_q + \frac{mB^2}{4(m-1)} \sqrt{\frac{\log(1/\alpha)}{2n}}.$$ Here, the independent sampling units are the $n$ retained-feature examples. For deployed probabilities, $B=1$ is valid; for effective logits in $[-K,K]$, $B=2K$. This also gives an explicit behavioral interpretation. For two independent replacements $T,T'\sim q$, $$\mathbb{E}_{Z,T,T'}\big[(f(Z,T) - f(Z,T'))^2\big] = 2D_q(f), \qquad T, T' \overset{\text{iid}}{\sim} q.$$ Therefore, on the same confidence event, for any tolerance $\tau>0$, $$\Pr_{Z,T,T'}\big(|f(Z,T) - f(Z,T')| > \tau\big) \le \min\Big\lbrace 1, \frac{2U_\alpha}{\tau^2}\Big\rbrace.$$ This is a computable statistical bound on prediction changes under the declared replacement mechanism. It requires neither optimizer convergence nor a Lipschitz constant. The audit must be independent of training and checkpoint selection, or selection must be accounted for through a simultaneous confidence bound. We will state this finite-sample construction explicitly. Its scope remains important: it certifies distribution-averaged dependence under $q$. Range-wide guarantees additionally require verified regularity bounds, and proximity to a retrained predictor requires the separate comparison addressed by Theorem 4.3 or a direct prediction-distance audit. We do not treat the existing RIPV point estimates as already supplying all these guarantees.

3. Narrow intervals provide interval-specific guarantees, but BMCR is not restricted to narrow intervals.

The reviewer is correct that a replacement interval such as [−ε,ε][-\varepsilon,\varepsilon] in standardized space need not cover the complete feature domain. The corresponding result therefore concerns that replacement distribution and interval. Standardization alone does not establish full-domain coverage.

This distinction is explicit in the paper. Definition 3.1 scopes the guarantee to supp⁡(q)\operatorname{supp}(q), Section 3.2 distinguishes known-bounds and interval-specific settings, and Corollary D.7 states that the guarantee does not extend outside the replacement support. Table 1 likewise distinguishes BMCR-ε\varepsilon from BMCR-full.

When valid feature bounds [ℓi,ui][\ell_i,u_i] are known, BMCR can instead use

Ti∼Unif⁡[ℓi,ui],T_i\sim\operatorname{Unif}[\ell_i,u_i],

which covers the entire valid domain by construction, without requiring the revoked feature's empirical distribution (Section 3.2 and Corollary D.6). Full-domain coverage does not itself establish a small finite-error certificate; the relevant dependence and regularity bounds must still be supplied. Similarly, empirical min–max bounds establish an observed range, not necessarily the full population domain.

The experiments also extend beyond the narrow interval highlighted in the review. Appendix Q evaluates replacement widths ε∈{0.1,0.5,1,5,10}\varepsilon\in\{0.1,0.5,1,5,10\}. More directly, the controlled experiment in Appendix E.1 has a known true feature domain:

\qquad T\sim\operatorname{Unif}[-1,1].$$ Replacement therefore covers the entire feature domain in that experiment. At $\rho=0.99$, after 200 epochs, the reported independent-replacement variance is $0.0029\pm0.0007$, the normalized prediction distance to retraining is $0.023\pm0.006$, and the test-loss increase relative to retraining is $0.78\pm0.21\%$ (Tables 6–7). These are empirical results under full-domain replacement, rather than a complete numerical range-wide certificate. We will make the reporting distinction explicit in the main text and experimental captions: each result will be interpreted under its stated replacement support; RIPV point estimates will be distinguished from statistical upper bounds; and full-domain numerical claims will require valid domain coverage and the additional verified conditions appropriate to the claimed guarantee. **W2: Cost at matched residual dependence.** Given that the epochs in Table 2 are checkpoints of a single run, the table directly gives the cost of reaching each RIPV level: | Epoch | RIPV | TRI | EI | | --- | --- | --- | --- | | 1 | 0.0241 | 0.931 | 2213.8 | | 100 | 0.0102 | 0.984 | 40.3 | | 200 | 0.0087 | 0.985 | 20.4 | | 600 | 0.0071 | 0.986 | 6.9 | | 1500 | 0.0059 | 0.987 | 2.7 | - *Every level is reached faster than retraining.* Retraining removes the input (RIPV = 0) at EI = 1 by definition, and EI is measured against the time retraining needs to reach its selected checkpoint, not its epoch cap. Every BMCR checkpoint is cheaper, including the lowest-dependence one at epoch 1500 (EI 2.7 on ResNet; 2.4 on MLP and FT-Transformer). - *Most of the reduction happens early.* Epoch 100 already achieves 76% of the RIPV reduction between epochs 1 and 1500 (0.0241 → 0.0102, vs. 0.0059 at epoch 1500), with the same utility (TRI 0.984 vs. 0.987), at about 15× lower cost than running to epoch 1500. - *The budget is an operating-point choice.* On MLP, utility keeps improving after epoch 100 (TRI 0.927 → 0.976, Appendix Q), so a longer budget is worthwhile there; it remains cheaper than retraining. On the source of the ~40× figure: it is the epoch-100 EI in Table 2, averaged over all seven datasets and both features, and the other backbones give similar values (34.0 ± 3.9 for MLP, 34.5 ± 3.8 for FT-Transformer; Appendix Q). The Magic/MLP timing in Section 5 (39.7×) is only an end-to-end check. **Q1: Non-zero RIPV for SFI and FPE.** We thank the reviewer for this careful check. The reviewer is correct: SFI and FPE cannot respond to $x_i$, so their RIPV must be exactly zero. There is no other input path, and all methods use the same RIPV definition and code. The non-zero values came from an evaluation error: RIPV was computed with the network in training mode, so dropout was active. Each forward pass then used a different dropout mask, and the variance across replacement draws $T_{jr}$ included dropout noise, which is present even for a model that ignores $x_i$.

Correction. We recomputed RIPV with the network in evaluation mode (dropout disabled) for all methods.

  • SFI and FPE now have RIPV exactly 0 on every dataset and backbone (Table 3 and Tables 43–49, formerly 42–48). As a check, changing only xix_i leaves the output of every SFI and FPE model unchanged.
  • The same noise had inflated BMCR's RIPV, which also decreases (Table 3: 0.011 → 0.008 for BMCR$1$ and 0.007 → 0.005 for BMCR${10}$). We applied the same correction to all RIPV values in the paper, including Table 2.

Q2: Obtaining numerical certificates for finite training checkpoints.

We thank the reviewer for asking how the theoretical statements can be instantiated numerically. The relevant distinction is between an excess-risk-dependent guarantee and a post-training certificate for a fixed predictor. The latter can be constructed from an independent audit without assuming that non-convex fine-tuning has reached the Bayes optimum. We explain how optimization quality, finite-sample uncertainty, and regularity enter this procedure.

1. Optimization error must be distinguished from an observable comparator gap.

The quantities γ\gamma and η\eta in Theorem 4.3 are population excess risks relative to the retained-feature Bayes predictor. They cannot be inferred merely from small training loss, a small gradient norm, or convergence of AdamW. We do not claim that the reported training curves provide certified numerical upper bounds on these quantities.

The quantity denoted εopt\varepsilon_{\mathrm{opt}} in Corollary E.1 has a more specific meaning: its premise is

\le \widehat R_q(g^{\mathrm{rt}}) + \varepsilon_{\mathrm{opt}},$$ where $g^{\mathrm{rt}}$ is a specified retrained comparator. Given both models, this premise can be checked using $$\varepsilon_{\mathrm{comp}} = \big[\widehat{R}_q(f) - \widehat{R}_q(g^{\mathrm{rt}})\big]_+.$$ This is an empirical comparator gap, not a certified distance to the global optimum. We will clarify this terminology. For deployment, a direct alternative is to certify the achieved residual dependence of the frozen predictor and, when a retrained reference is available, its actual prediction distance to that reference. Optimization quality is then reflected in the observed quantities being bounded, without requiring an estimate of the unknown Bayes optimization gap. **2. Finite-sample error can be bounded on an independent audit set with a specified confidence level.** Fix the trained checkpoint, its inference pipeline, and the replacement distribution $q$ before auditing. Let $Z_j=X_{-i,j}$, $j=1,\ldots,n$, be fresh independent audit examples drawn from the relevant deployment distribution. For each example, draw $m\ge2$ independent replacements $T_{jr}\sim q$. Suppose the deployed scalar output lies in a known interval of width $B$. This condition holds with $B=1$ for probabilities and $B=2K$ for effective logits in $[-K,K]$. The bound must refer to the output actually being audited; an observed sample range is not a substitute for a known output bound. Using the unbiased within-example variance, define $$\overline{F}_j = \frac{1}{m}\sum_{r=1}^{m} f(Z_j, T_{jr}),$$ $$\widehat{D}_q = \frac{1}{n}\sum_{j=1}^{n} \frac{1}{m-1}\sum_{r=1}^{m} \big[f(Z_j, T_{jr}) - \overline{F}_j\big]^2.$$ Then $\mathbb E[\widehat D_q]=D_q(f)$. Each within-example variance is bounded by $$C_m=\frac{mB^2}{4(m-1)}.$$ Consequently, with probability at least $1-\alpha_D$ over the audit sampling, $$D_q(f) \le U_D := \min\Big\lbrace \frac{B^2}{4},\ \widehat{D}_q + C_m\sqrt{\frac{\log(1/\alpha_D)}{2n}} \Big\rbrace.$$ This gives an explicit finite-sample upper confidence bound on the residual dependence of the deployed checkpoint. It requires neither a Bayes-risk estimate nor a Lipschitz constant. The independent sampling units are the $n$ retained-feature examples, not the $nm$ predictions, because repeated replacements share the same retained input. These are fixed-predictor bounds on fresh audit data. They should not be confused with a numerical bound on the class-wide supremum $\Delta_{n,m}$ appearing in Corollary E.1. Using an independent audit provides a separate route that avoids estimating that supremum.
Reviewer_QudBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a method for output-level certified unlearning of features. Output-level means that the deviation between the unlearned model's predictions (outputs) and those of the retrained model without access to the removed feature(s) is controlled with high probability.

The method is simple: replace the feature-to-forget with noise defined over some range I\mathcal{I}, then finetune your base model on the updated predictors. Assuming that this finetuning produces a near-optimal model, then the dependence on the removed feature would be controlled with high probability.

Critical strengths

S1. The proposed method does not require or assume access to the removed feature, a setting that is practical in real-life scenarios. It is simple and substantial effort is put to motivate it theoretically, with a series of theoretical results certifying the output-level unlearning indistinguishability from a retrained baseline.

Critical weaknesses

Despite the simplicity of the proposed method and the practicality of the problem statement, the paper is poorly written and contains critical issues which make me recommend rejection. An adequate rebuttal would make me happily revisit my recommendation.

W1. The paper is poorly written. In multiple places, symbols are used without definition. The Method section starts abruptly without any explanation. This is present in the following instances:

  • i) In line 130, xx, x−ix_{-i} must be defined. Are these real valued? Is xx a random variable or an instance? What probability distribution is it distributed according to (if any)?
  • ii) Line 134, what does a "missing coordinate" even mean? One could argue that instead of unlearning it, we can use imputation to predict it. You, most probably, mean "coordinate-to-forget" or something. Please be precise and clear.
  • iii) Line 138, the measure Px−iP_{x_{-i}} must be defined, and under which a probability distribution of the data or the random range (e.g., qq) must also be specified. Otherwise, the readers won't be so clear where the data, features, labels, or replacements come from. Please consider removing the abruptness in the use of the notation. What is fθ(x−i,t)f_{\theta}(x_{-i},t)? It seems to be the prediction of the model when xix_i is replaced by tt but it is not defined anywhere.
  • iv) Line 139, you say "If, in addition, t↦fθ(x−i,t)t \mapsto f_{\theta}(x_{-i}, t) is continuous on supp(q), then this q-almost-sure invariance extends to pointwise invariance over supp(q)." This is ambiguous or wrong. There is no reason for the almost-sure invariance to imply pointwise invariance even if the mapping is continuous. Another restatement of this appears in line 154.
  • v) Line 172, the authors introduce two cases of BMCR: the first is of a uniform replacement; the second is of a mean-centered replacement. There is no discussion of why the mean-centered is preferred over the uniform replacement. If this mean-centered case (or the uniform case thereof) does not support or add any value, then no need to include it. It simply inflates the exposition without any substance. If the advantage of either of these cases is given in the appendix, then please refer to that appendix in the main text.
  • vi) Line 187, in my opinion, there is no need for a full paragraph on the finetuning objective. This is a standard regularised empirical risk minimisation. Dedicating a full paragraph for it gives a wrong impression of an important contribution when it is actually not. A concise mention of the type of objective you are trying to minimise should suffice.
  • vii) Line 226, what does it mean for YY to be L2L_2? This needs a clarification before it is used inside a theorem.
  • viii) The figures and tables are impossible to read. Showing extremely vague figures and tables cannot support the made arguments in any meaningful way. Table 2 is so hard to read and its corresponding discussion starting at line 415 is extremely ambiguous, and there is no explanation or discussion why the other baselines behave as reported. The caption is not informative and it is not clear what the header row shows. That cannot be left to the appendix, it must be in the main text. Figure 2 gives no clear distinction between lines. It is challenging to know which one improves over which. Table 3's caption is poorly written and it is not clear why you would average over "three architectures" and why there is no std in Table 3(a). The whole discussion under the EXPERIMENTS AND RESULTS section needs considerable improvement. Are the authors sure this is how they would love to present their experiments?

W2 [CRITICAL]. All theoretical results rest heavily on the assumption that, after finetuning, the unlearned model (the model finetuned after the feature replacement) is near-optimal to the Baye's predictor. This is infeasible in practice unless strong assumptions hold, which is not the case in deep learning applications (the setting of the paper since the experiments use deep neural network architectures). This undermines the contribution of the paper heavily.

  • i) For this method to be of practical interest, it has to achieve near-optimal minimisation. This cannot be an assumption. It must be shown to be achievable or otherwise the whole work becomes superficially appealing but internally empty.
  • ii) The Shuffle method (Yang et al. 2025) uses the available feature values in the data to shuffle them. If the number of data points goes to infinity in the limit, we know that the available feature values frequencies would approximate the actual distribution. In that case, shuffling them is nothing but randomly sampling from the range on which the feature-to-forget is supported under the estimated probability distribution. This seems to converge to the proposed method. This deserves a discussion. I understand that this still requires access to the actual feature values and that an infinitely large dataset is unpractical, but the theoretical comparison would be of interest to the reader.
Questions for authors

Q1. Please explain all used notation before it appears in a result or discussion. (W1 i, ii, iii, & vii)

** Q2.** Can you please clarify how almost-sure invariance implies pointwise invariance when the mapping t↦fθ(x−i,t)t \mapsto f_{\theta}(x_{-i},t) is continuous? If it is not the case, then simply remove it as I don't see how it affects your whole argument. (W1 iv)

Q3. Can you explain how the mean-centered is preferred over the uniform replacement of BMCR or vice versa? (W1 v)

Q4. Please consider completely redesigning and repositioning the figures and tables to clarify the results. (W1 viii)

Q5. Please distinguish between the assumption that the unlearned model is δ−optimal\delta-optimal and a proof that it is as such. This means that without a valid proof of δ−optimality\delta-optimality, there is no reason to believe that the proposed approach is actually better than any other approach under the same assumption. (W2 i)

Q6. Please highlight the link of BMCR and that of Shuffle when the number of data samples goes to infinity. (W2 ii)

9 comments

We are grateful to the reviewer for their rigorous and constructive review. We address each point below:

Weakness1 (i)–(vii): notation and exposition.

We thank the reviewer for the careful reading. We have rewritten Section 3 and the affected parts of Section 4 (changes in blue). Below we answer each point directly.

(i) What are x−ix_{-i} and xix_i?

  • Data are pairs (X,Y)∼P(X,Y)\sim P, where PP is the unknown data distribution, X=(X1,…,Xd)∈RdX=(X_1,\dots,X_d)\in\mathbb R^d is the real-valued feature vector, and Y∈RY\in\mathbb R (regression) or Y∈{0,1}Y\in\lbrace 0,1\rbrace (classification).
  • We write X=(X−i,Xi)X=(X_{-i},X_i), where Xi∈RX_i\in\mathbb R is the revoked feature and X−i∈Rd−1X_{-i}\in\mathbb R^{d-1} are the retained features. These are random variables; lowercase x=(x−i,xi)x=(x_{-i},x_i) denotes a realization, i.e., one observed feature vector. The training set {(x(j),y(j))}j=1n\lbrace(x^{(j)},y^{(j)})\rbrace_{j=1}^n consists of i.i.d. draws from PP.

(ii) "Missing coordinate."

We meant revoked: the values of xix_i exist but may no longer be used (e.g., because the feature has become legally restricted), so recovering them is not the goal. Imputation is not suitable here for three reasons:

  1. Fitting an imputer of xix_i from x−ix_{-i} requires joint access to the revoked values, which is exactly what is no longer permitted.
  2. Feeding an imputed value to the model re-inserts a reconstruction of the revoked attribute into every prediction, which is the opposite of removing its influence.
  3. Even with a perfect imputer, plugging x^i=E[Xi∣x−i]\hat x_i=\mathbb E[X_i\mid x_{-i}] into the original model gives fθ0(x−i,x^i)f_{\theta_0}(x_{-i},\hat x_i), which in general differs from the retrained target E[Y∣X−i=x−i]\mathbb E[Y\mid X_{-i}=x_{-i}] because fθ0f_{\theta_0} is nonlinear in xix_i.

The goal is instead a model whose predictions do not depend on coordinate ii at all while preserving the predictive relationship between X−iX_{-i} and YY, which is the behavior of a model retrained from scratch without xix_i. "Missing" has been replaced by "revoked" throughout.

(iii) Probability measures and fθ(x−i,T)f_\theta(x_{-i},T).

TT is a random replacement value drawn from a distribution qq on R\mathbb R chosen by the model owner (e.g., q=Unif[a,b]q=\mathrm{Unif}[a,b]), independently of (X,Y)(X,Y). As the reviewer suggests, fθ(x−i,T)f_\theta(x_{-i},T) is the model's prediction when the revoked value is replaced by TT; more generally, f(x−i,t)f(x_{-i},t) is the prediction when the value tt is placed in coordinate ii. The residual dependence is

Dq(f)=EX−i∼PX−i[VarT∼q(f(X−i,T))],D_q(f)=\mathbb E_{X_{-i}\sim P_{X_{-i}}}\Big[\mathrm{Var}_{T\sim q}\big(f(X_{-i},T)\big)\Big],

where PX−iP_{X_{-i}} is the marginal law of the retained features under PP. The inner variance is over T∼qT\sim q with x−ix_{-i} held fixed, and the outer expectation averages over X−iX_{-i}. Thus Dq(f)=0D_q(f)=0 if and only if, for almost every x−ix_{-i}, the prediction does not change when coordinate ii is resampled from qq. The replacement risk is defined on the same probability space,

Rq(f)=E(X−i,Y)∼PET∼q[ℓ(f(X−i,T),Y)],R_q(f)=\mathbb E_{(X_{-i},Y)\sim P}\mathbb E_{T\sim q}\big[\ell\big(f(X_{-i},T),Y\big)\big],

and for a model gg that uses only the retained features (e.g., one retrained without xix_i), R−i(g)=E(X−i,Y)∼P[ℓ(g(X−i),Y)]R_{-i}(g)=\mathbb E_{(X_{-i},Y)\sim P}[\ell(g(X_{-i}),Y)]. We write Dq(θ)=Dq(fθ)D_q(\theta)=D_q(f_\theta) and Rq(θ)=Rq(fθ)R_q(\theta)=R_q(f_\theta), and qiq_i, TiT_i when the coordinate needs emphasis, as in Theorem 4.2.

(iv) Almost-sure vs. pointwise invariance.

We respectfully believe the statement is correct once supp(q)\mathrm{supp}(q) is understood as the (closed) support of qq,

supp(q)={t∈R: q((t−r,t+r))>0 for every r>0},\mathrm{supp}(q)=\lbrace t\in\mathbb R:\ q((t-r,t+r))>0\ \text{for every } r>0\rbrace,

i.e., the smallest closed set of qq-probability one. We agree that this definition and the argument should have been given, and the revision now provides both.

Claim. If g:supp(q)→Rg:\mathrm{supp}(q)\to\mathbb R is continuous and g(T)=cg(T)=c qq-almost surely, then g(t)=cg(t)=c for every t∈supp(q)t\in\mathrm{supp}(q).

Proof. Let A={t∈supp(q):g(t)=c}A=\lbrace t\in\mathrm{supp}(q): g(t)=c\rbrace, so q(A)=1q(A)=1. Fix t0∈supp(q)t_0\in\mathrm{supp}(q). For every r>0r>0, q((t0−r,t0+r))>0q((t_0-r,t_0+r))>0 and q(Ac)=0q(A^c)=0, so (t0−r,t0+r)∩A≠∅(t_0-r,t_0+r)\cap A\neq\emptyset. Hence there are tn∈At_n\in A with tn→t0t_n\to t_0, and continuity gives g(t0)=lim⁡ng(tn)=cg(t_0)=\lim_n g(t_n)=c. □\square

Application. If Dq(f)=0D_q(f)=0, then VarT∼qf(x−i,T)=0\mathrm{Var}_{T\sim q}f(x_{-i},T)=0, i.e., f(x−i,T)=fˉ(x−i)f(x_{-i},T)=\bar f(x_{-i}) qq-almost surely, for PX−iP_{X_{-i}}-almost every x−ix_{-i}, where fˉ(x−i)=ET∼q[f(x−i,T)]\bar f(x_{-i})=\mathbb E_{T\sim q}[f(x_{-i},T)]. Applying the claim to g=f(x−i,⋅)g=f(x_{-i},\cdot) gives f(x−i,t)=f(x−i,t′)f(x_{-i},t)=f(x_{-i},t') for all t,t′∈supp(q)t,t'\in\mathrm{supp}(q), for PX−iP_{X_{-i}}-almost every x−ix_{-i}. The conclusion holds only on supp(q)\mathrm{supp}(q), which is why all our guarantees are stated over the support.

Why the assumptions are needed. Continuity cannot be dropped: for q=Unif[0,1]q=\mathrm{Unif}[0,1], g=1{t=1/2}g=\mathbf 1\lbrace t=1/2\rbrace satisfies g(T)=0g(T)=0 almost surely but g(1/2)=1g(1/2)=1. The support must also be the closed support rather than an arbitrary set of probability one, since the argument uses that every neighborhood of a support point has positive mass.

In the revision, supp(q)\mathrm{supp}(q) is defined where it first appears, the argument is stated as a lemma with proof in the appendix, and the two restatements (old lines 139 and 154) are merged into a single statement that cites it.

(v) Uniform vs. mean-centered replacement.

Our exposition obscured that these are not competing variants. The certificate depends only on the support of qq, not on its mean: the bound Dq(f^)≤Rq(f^)−inf⁡gRq(g)D_q(\hat f)\le R_q(\hat f)-\inf_g R_q(g) (Theorem 4.2) holds for any qq drawn independently of (X,Y)(X,Y), and it certifies invariance on supp(q)\mathrm{supp}(q). Hence any q=Unif[a,b]q=\mathrm{Unif}[a,b] whose support covers the interval to be unlearned is valid, and the two cases differ only in what information about xix_i is needed to choose [a,b][a,b]:

  1. Declared support. If valid bounds [ℓi,ui][\ell_i,u_i] are known from the schema or the deletion request (e.g., a binary indicator, an age range, or a percentage), q=Unif[ℓi,ui]q=\mathrm{Unif}[\ell_i,u_i] certifies unlearning over the full feature domain.
  2. Standardized range. Tabular models usually consume standardized inputs (xi−μi)/σi(x_i-\mu_i)/\sigma_i, where (μi,σi)(\mu_i,\sigma_i) are scaler parameters already embedded in the deployed pipeline. Sampling T∼Unif[−ε,ε]T\sim\mathrm{Unif}[-\varepsilon,\varepsilon] in the model's own input space, as in our experiments, computes no new statistic and certifies the raw interval [μi−εσi, μi+εσi][\mu_i-\varepsilon\sigma_i,\ \mu_i+\varepsilon\sigma_i].

The second interval is centered at the training mean only because the scaler is. Centering does not affect the certificate; its practical benefit is that replaced inputs stay in the region the network was trained on, which avoids a systematic input shift when fine-tuning starts. We removed the general center/radius derivation from the main text and now present these two cases directly, with a pointer to the appendix remark on centering.

(vi) Fine-tuning objective.

We agree that the objective is standard regularized risk minimization of RqR_q, with a fresh draw of TT for every example in every mini-batch, and it is not a contribution in itself. We replaced the paragraph and its three equations with a short paragraph. We kept a single line on how weight decay affects the guarantee: if θ^\hat\theta minimizes Rq(θ)+λ2∥θ∥22R_q(\theta)+\frac{\lambda}{2}\lVert\theta\rVert_2^2, then for any reference parameter θˉ\bar\theta,

Rq(θ^)−inf⁡gRq(g)≤Rq(θˉ)−inf⁡gRq(g)+λ2∥θˉ∥22,R_q(\hat\theta)-\inf_g R_q(g)\le R_q(\bar\theta)-\inf_g R_q(g)+\tfrac{\lambda}{2}\lVert\bar\theta\rVert_2^2,

so the certificate degrades only additively.

(vii) What does Y∈L2Y\in L_2 mean?

It means E[Y2]<∞\mathbb E[Y^2]<\infty. This ensures that squared-loss risks are finite for square-integrable predictors and that the Bayes predictor E[Y∣X−i]\mathbb E[Y\mid X_{-i}] minimizes them. Theorem 4.2 now states the condition explicitly as E[Y2]<∞\mathbb E[Y^2]<\infty, together with the predictor class (measurable ff with E[f(X−i,T)2]<∞\mathbb E[f(X_{-i},T)^2]<\infty). We also define L2(μ)L_2(\mu) for a probability measure μ\mu, with norm ∥g∥L2(μ)=(Eμ[g2])1/2\lVert g\rVert_{L_2(\mu)}=(\mathbb E_\mu[g^2])^{1/2}, which is the norm used in Theorem 4.3.

viii) Figures, tables, and the Experiments section. We have rewritten Section 5 and redesigned its tables and figure.

Table 2 and its discussion.

  • Header row. The columns 1, 10, ..., 1500 are unlearning epochs, i.e., checkpoints of one 1500-epoch BMCR run; the header now says "Unlearning epoch".
  • Readability. Each metric shows its ideal direction (e.g., RIPV ↓\downarrow, TRI →1\to 1, MIA →0.5\to 0.5), rows are grouped (certificate; utility and cost; dependence audits; privacy and fairness), the epoch-100 operating point is shaded, and cell spacing is fixed. The "Orig." column, which held a single value, is removed; that value is in the caption.
  • Caption. It states what each cell reports (mean and std over 7 datasets ×\times {most, least influential feature}), what the std captures (differences between tasks, not run-to-run noise), why ResNet with ε=0.1\varepsilon=0.1 is shown (the narrowest range, hence the weakest replacement signal), and where other backbones and ranges are.
  • Placement and discussion. The table now follows the metric definitions, and the paragraph formerly at line 415 reads it explicitly: left to right is one run over time, with epoch 100 as the shaded operating point.

Why the baselines behave as reported. We agree this belongs in the main text and added a paragraph explaining each baseline by what it substitutes for xix_i:

  • FA uses a constant, i.e., a point-mass qq, for which Dq(f)=0D_q(f)=0 for every ff, so the objective puts no pressure on the xix_i pathway. This explains its utility loss (TRI 0.64–0.84 on MLP/ResNet) and remaining dependence (FA0: RASI 0.61–0.71). SAM samples a narrow window around the mean and behaves like FA0.
  • GS, BMCR-MM, and Shuffle draw xix_i from a spread-out distribution, the same mechanism as BMCR, and perform similarly (TRI ≈1\approx 1, RASI 0.96–0.99), but need statistics or per-example values of xix_i; BMCR-ε\varepsilon needs neither.
  • MIUL removes mutual information from the representation rather than the output's sensitivity to xix_i, so it keeps utility but still uses xix_i (RASI 0.53–0.56).
  • FOUL loses utility on every backbone (TRI 0.65–0.67); its near-perfect invariance on FT-Transformer comes with that loss, so invariance metrics must be read together with TRI.

Figure 2.

  • Methods are grouped into visual families: BMCR in shades of one blue (ε\varepsilon = 0.1, 1, 10, light to dark) as thick solid lines on top; full-access baselines dashed; replacement baselines using data statistics thin solid; fixed-value baselines gray dotted.
  • Each method has its own marker, so lines stay distinguishable without color; the palette passes a colorblind-safety check. Shaded bands now match their line's color.
  • The epoch axis is logarithmic, so the early epochs where most change happens are no longer compressed; the EI axis is logarithmic, so high-EI methods no longer flatten the others.
  • Each column title gives the ideal value, marked by a labeled dashed line (for EI, the line marks retraining cost; higher is faster). The caption explains the encoding.

Where lines still overlap, the values are nearly identical (e.g., GS, BMCR-MM, and BMCR$_1$ for TRI on MLP); exact numbers are in Tables 18–20.

Table 3. The original table combined two unrelated experiments under one caption. It is now split into Table 3 (structural masking vs. BMCR) and Table 4 (attribute inference), each with its own caption.

  • Std. Table 3 now reports mean ±\pm std across the seven datasets, computed from the per-dataset results in Tables 43–49.
  • Why average over three architectures. SFI and FPE modify only the input layer, so the comparison concerns the removal mechanism rather than the backbone; the caption now says so, and Tables 43–49 report the variation over backbones and seeds.
  • The caption also states that all methods use no information about xix_i and the same fine-tuning budget, so their cost is equal and EI is omitted.

We also rewrote the structural-masking paragraph to state what the comparison shows: SFI and FPE have RIPV exactly 0 by construction but stay further from retraining (TRI 0.78 and 0.90 vs. 0.99–1.01 for BMCR).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.