acceptodds
Withdrawn from ICLR 2027

Auditing Static Occlusion Ordering Under Prediction-Deviation Budgets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

After making a prediction, a data holder may want to release or store fewer fields while keeping that prediction approximately unchanged. One way to choose fields for deletion is to measure their individual effects once and try the least sensitive first. The difficulty is that every accepted deletion changes the context for later trials, and measuring sensitivities consumes queries that could instead explore another path. We audit this trade-off in a cost-tiered deletion procedure that checks each removal against the original prediction. On taxi and movie data, static sensitivity ordering reduces retained cost relative to alphabetical ordering and two random paths given the same query allowance. Under the baseline mask, its relative RMSE increase stays below the chosen tolerance over the first four tested budgets; wider budgets permit larger savings but can exceed that tolerance. Repeated random paths and cost reassignments largely preserve the advantage, whereas alternative masking can reverse it. Static sensitivities therefore provide useful ordering information in these settings, with the benefit depending on the allowed prediction drift and the representation of deleted fields.

Primary area:
interpretability and explainable AI
Keywords:
feature occlusion, contextual disclosure, attribution ordering, tabular machine learning, prediction deviation, privacy-aware data minimization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_upuYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies sequential field deletion under a bound on deviation from the original prediction. Fields are ranked once by their individual prediction effects, then tested for deletion within descending cost tiers. Experiments on taxi and movie data compare retained cost and prediction error against lexical, random, and dynamically updated orders.

Critical strengths
  1. The question is specific. Field deletion is written as an audited procedure under a query budget, where each deletion is checked against the original prediction and the queries spent on scoring are counted.
  2. The paper reports the boundary of its claim. Prediction error stays within tolerance over the first four budgets, wider budgets save more but can exceed it, and a different masking of deleted fields can reverse the advantage.
Critical weaknesses
  1. The reported results do not establish an efficiency advantage of static scoring over dynamic updating. Dynamic updating lowers the retained cost in 18 of 20 conditions and uses 15 and 14 queries on NYC and IMDb against 16 and 14 for static ordering. The paper acknowledges this limitation, but does not establish under what conditions static scoring would offer a better trade-off than dynamic updating.
  2. The comparison also depends on a rule that forbids retries. Each field is tried once, and a rejected field is never tried again. The paper's own example shows the rule can decide the result: the lexical order rejects b, a second try would delete b, but the rule forbids it. Whether static ordering still wins when retries are allowed is not reported.
Questions for authors
  1. Since dynamic updating is no more expensive in the reported runs, under what conditions does computing the scores once do better?
  2. How often would a second try have succeeded in the reported runs? Would allowing retries change the comparison between orderings?
  3. Dynamic updating wins in 18 of 20 conditions. Which two conditions favour static ordering, and how large is the difference there?
Reviewer_hcFNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for safely deleting fields from a database while keeping the prediction from a Machine Learning model approximately unchanged. Experimental evidence supports the use of the proposed approach.

Critical strengths

None.

Critical weaknesses

I honestly see several weaknesses in this work.

  • w1: motivation I do think this paper is not properly framed. The main reason I see for this work is privacy, but the work seems rather dismissive in that regard. I think a better motivation for the work is mandatory.

  • w2: presentation The work is poorly written, making it very difficult to understand what is going on. Several terms are used as if they must be known, but are not given proper definitions. E.g., while I can imagine what the authors mean by policy, they never define what a policy is in this setting. Moreover, some parts of the text are very confusing: e.g., lines 191-196 seem to refer to something that is not in Figure 1. This makes it very hard to understand what the authors are talking about. I would suggest that the authors restructure the paper and clearly distinguish the background section, the methodology section and the experimental section, divided into experimental settings (where they can describe metrics, datasets and methods) and experimental results. This would make (in my opinion) the workflow better.

  • w3: soundness The work does not consider any theoretical analysis of the proposed task. As far as I can see, the problem presented in Eq. 1 is not tractable. Thus, I would have expected some theoretical analysis (e.g., the complexity of the task or some guarantees) of the proposed heuristics. Such an analysis is not provided in the main paper, and I do not think that explicitly mentioning the lack of approximation guarantees in the Limitations solves my concern. Moreover, several empirical choices must be better motivated (e.g., why were the fields, costs, and masking chosen at those values?). Finally, I think a very natural baseline ordering features using their Shapley Values is missing.

  • w4: novelty In [1], the authors investigate a very similar problem: how to select features to keep per individual while preserving an ML model prediction at inference time. I think a discussion/comparison is mandatory.

I think these four points strongly prevent me from recommending acceptance in the current state of the work.

[1] - C. Tran, F. Fioretto. Data Minimization at Inference Time. NeurIPS 2023

Questions for authors

Please address the highlighted weaknesses in the work.

Reviewer_vjAsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Given a model ff that takes nn features as input, the authors aim to occlude a subset of features while still retaining predictive performance. They compare their solution---a threshold on a greedy sensitivity scoring---with two baselines: selecting features based on a lexicographical ordering of the field names and random selection.

Critical strengths

None

Critical weaknesses

The current exposition does not meet the expected quality of an ICLR publication.

Additionally, the main contributions are meaningless:

  • They show that their feature occlusion method is better than random and than an arbitrary lexicographical ordering;

These are not reasonable baselines.

  • The second contribution is borderline unintelligible:

"We relate the budget-dependent burden reduction to the method’s own predictive utility and examine how masking choices change the ordering comparison."

Burden is only defined on page 5, and predictive utility only appears again in the supplementary material, with no definition. There is an item named Utility degradation. in Sec 5.1, but it defines RMSE and only mentions utility.

Questions for authors

Could you clarify your second contribution?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.