acceptodds
Withdrawn from ICLR 2027

SHERLOC: Statistical Hypothesis Evaluation and Recovery for Learning Opponent Characteristics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

11 views

Abstract

When used in automated decision-making systems, machine learning (ML) models are vulnerable to data-manipulation attacks. Some defense mechanisms (e.g., adversarial regularization) directly affect the ML models while others (e.g., anomaly detection) act within the broader system. However, many defenses operate under a predefined threat model that assumes characteristics of the adversary, including their knowledge, capabilities, and objectives, are known and fixed. In practice, these characteristics may be unknown or evolve over time, creating a mismatch between the assumptions of the defender and the true attacker. In this paper, we study a new perspective on adversarial defense: focusing on the attacker, rather than the attack. We present and demonstrate SHERLOC, a framework for adversarial learning forensics – namely, a detective that learns about the attacker from observed attack(s). We prove that, without additional knowledge, the attacker is non-identifiable — multiple potential attackers would perform the same observed attack. This reveals an important distinction between reconstructing an attacker's behavior and recovering the attacker that generated it. We demonstrate empirically that SHERLOC is able to reconstruct a hypothesis attacker that is consistent with the observed attacks against various defender models (i.e., generalized least-squares linear regressors, logistic regressors, and polynomial classifiers) across various security-critical domains. Finally, we show that these inferred attackers need not match the ground truth attacker exactly to have downstream utility. Defenses using the SHERLOC-inferred attackers achieve approximately of robustness improvement that would be achieved if the defender had access to the ground-truth attacker.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
adversarial learning, trustworthy AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VbczOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SHERLOC, a framework for inferring an attacker's knowledge, capabilities, and objectives from observed adversarial attacks. The authors formulate attacker inference as an inverse optimization problem and evaluate attack reconstruction, parameter recovery, and downstream defense on linear and polynomial models.

Critical strengths
  1. The goal of this paper is interesting how do we know the attack's ability is an important problem
  2. The distinction between attack reconstruction and attacker recovery provides an interesting perspective on adversarial forensics.
Critical weaknesses
  1. Limited significance and practical value. The related work is insufficiently discussed, and the paper lacks a convincing real-world application scenario. The evaluation is limited to simple models, without considering modern deep learning models or LLMs, substantially limiting its significance.

  2. Overly restrictive assumptions. The defender is assumed to know both the adversarial and original inputs, which is impractical in realistic settings. Moreover, the defender need know the attacker's parameterized form in advance.

  3. Unclear methodology and insufficient experiments. The methodology is not clearly explained in the main text, and the experiments lack comprehensive comparisons with relevant existing methods and practical baselines.

Questions for authors

See weaknesses

Reviewer_jYehOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper motivates that existing adversarial defenses for machine learning models assume a pre-defined threat model (knowledge, capabilities, objectives), and they do not reflect real-world scenarios. Because there may be assumption mismatch (unknown characteristics or evolving attacker). If the real attacker does not match the assumptions, the defense offers little or ineffective protection. To this end, the paper reframes the scenario. Instead of assumption, the new framing learns the attacker from the observed attacks. The paper makes the distinction between recovering the attacker and reconstructing the attacker. The paper shows that the exact recovery is limited and proposes SHERLOC to reconstruct the attack via optimization. Consequently, the inferred attacker parameters are applied into attacker-specific adversarial regularizer. Experiments are done on simple learners (linear and polynomial models) and show downstream defense applicability.

Critical strengths

The paper motivates the defense scenario to continually improve. Reconstructing behaviorally equivalent one is interesting, but conceptual. Connecting to reverse engineering of deception is also interesting. The paper shows the inferred attacker improves the defense, not just estimating the parameters. Overall, the framing is interesting. The paper brings more on conceptual/foundational contribution.

Critical weaknesses

The paper does not seem a deployable defense. The paper says no prior knowledge (other than the parameterization). Will the parameterization be some sort of assumption? Practically, the defender knows only the input from the attacker. The SHERLOC also assumes the defender knows the clean inputs. The reviewer thinks that is also not practical. If the attacker knows the defender, the attacker may intentionally lunch misleading attacks. One inferred attacker may not generalize the unlimited types of attack. The paper also misses a worse-case adversarial training baseline.

Questions for authors

How many observations are required to infer an attacker?

If the SHERLOC fits the wrong attack family, will it still recover knowledge, capability, and goal?

Reviewer_2y1XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Most defenses assume a fixed threat model, and perform poorly when the real attacker differs from the assumed one. This paper asks what a defender can instead infer about an attacker from the attacks it has already observed, modelling the attacker as a tuple of knowledge, capability, and objective. The authors first prove that this tuple is not identifiable: for any attack and any two fixed components, a third can be chosen that makes the same attack optimal. They then propose an optimisation procedure that infers an attacker hypothesis consistent with the observations, and evaluate it under three separate lenses — whether it reproduces the observed attacks, whether it recovers the true parameters, and whether it helps a downstream defense. The central finding is that these come apart: reconstruction can be near-perfect while recovery is poor, and defensive benefit tracks the former.

Critical strengths

Proving non-identifiability before proposing an inference method is the right order of operations, and it changes how the results should be read. A great deal of work in this area reports parameter-recovery numbers as though recovery were the obvious success criterion. This paper establishes first that an observed attack identifies an equivalence class rather than a point, then designs its evaluation around that fact by separating reconstruction from recovery. The theorem is proved only in the linear setting, but within that setting it is a strong statement — not a constructed counterexample but a result holding for any attack and any two fixed components.

The paper's central empirical finding follows from that separation and is genuinely counterintuitive in a useful way. On the linear pendigits setting, knowledge recovery is essentially zero while the inferred attacker captures nearly all of the defensive benefit available from knowing the ground truth. That single result makes the paper's argument concrete: a defender does not need to identify who the attacker is in order to learn enough about how they attack to respond. I also want to credit the decision to report strongly negative recovery scores in the main tables rather than relegating them, since those numbers are what make the separation visible.

Critical weaknesses

The experimental scope is too narrow to support the generality the framing claims, and this is what drives my recommendation. The paper opens by motivating the problem with adversarial examples in computer vision and cites the deep-learning literature, but evaluates only linear regression, logistic regression, and degree-two polynomial classifiers on three small tabular datasets and synthetic problems up to sixteen dimensions. The non-identifiability theorem is likewise established only for linear defenders. This leaves the reader unable to judge whether the paper's headline claim — that reconstruction suffices for defense even when recovery fails — reflects something about adversarial learning or is an artifact of settings where a fixed closed-form optimal attack exists, a property the paper itself notes for linear learners. I do not think the fix requires scaling to ImageNet. A single small neural-network defender, an MLP on MNIST or a comparable tabular model, evaluated under the same three lenses, would tell the reader whether the reconstruction–recovery gap persists where the attack must be found by iterative optimisation rather than solved for. That one addition would materially change my assessment, and the existing framework appears general enough to admit it without redesign.

The second concern is that the evidence for the paper's most important claim is harder to assess than it should be, partly for presentational reasons and partly because of how settings are selected. The downstream-defense analysis is restricted to settings with measurable headroom, capability-based defense is dropped entirely because its headroom falls below a threshold, and what remains is five rows drawn from a small number of datasets. Because the excluded settings are not shown with their headroom values, a reader cannot see the shape of the selection. One reported gain exceeds one, which needs an explanation. Meanwhile the main tables carry blank cells attributed to experimental design, and reconstruction scores appear for some targets but not others within the same row, so no single configuration can be read end-to-end across all three lenses. All of this is fixable without new experiments: report every evaluated setting with its headroom so the selection is visible, consolidate into one table per learner with all three lenses populated or an explicit note where a cell was not run, explain the above-one gain, and add variance across trials. The underlying evidence may well be stronger than it currently appears.

Questions for authors

The question most likely to change my rating is about scope: do you have, or could you add, results for even one non-linear, non-polynomial defender such as a small neural network? I am asking specifically whether the reconstruction–recovery separation survives when the attacker's optimal attack is not available in closed form, since that is the regime the paper's motivation invokes.

I have three clarifications. First, in the downstream-defense table, what are the headroom values for the settings that were evaluated but not reported, and how many such settings are there? Second, how does a defense gain above one arise — is this measurement noise, or an interaction between the inferred attacker and the regularizer that lets the inferred hypothesis outperform ground truth? Third, do the reported scores reflect single runs or averages, and is there variance information available for the headline reconstruction and recovery numbers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.