acceptodds
Under review as a conference paper at ICLR 2027

-Audits: Adaptive Performance Monitoring

Abstract

Evaluations are the cornerstone of ML governance. Audits of ML-enabled systems are conventionally treated by regulators and academics as one-shot events: a single audit is conducted, its outcome is certified, and the system is presumed compliant until a fresh audit is commissioned. However, deployed systems are updated regularly, often without the auditor's knowledge, which can invalidate the audit's conclusions. To help regulators choose when to commission a fresh audit, we introduce \deltaaudits, a setting that pairs the initial white-box audit with a subsequent lower-access black-box monitoring phase. We instantiate monitoring with a fixed-budget disagreement test parameterized by a query distribution. We use the initial white-box inspection phase to optimize this distribution through a bi-level formulation that anticipates a provider maximizing its utility under the monitoring constraint. The goal is to limit deterioration in auditor utility while leaving the provider room to improve its own utility. Experiments on tabular and image classification tasks evaluate manipulation gain under uniform and optimized monitoring constraints. Optimized monitoring detects harmful updates with an order of magnitude fewer queries than uniform or performance-based monitoring, and sharply reduces the utility a provider can gain by manipulating the audit, whereas uniform monitoring leaves this gain almost intact.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI audits, performance monitoring, evaluation manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z97cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Δ-Audits, an adaptive performance monitoring framework designed to detect harmful model updates after an initial white-box audit. The framework combines an initial white-box inspection phase with a subsequent query-limited black-box monitoring phase, allowing auditors to assess whether deployed models have deviated from their previously audited versions. The authors introduce Twister, a bi-level optimization algorithm that constructs an optimized monitoring query distribution by anticipating a rational model provider's utility-maximizing behavior under monitoring constraints. By strategically assigning query probabilities to audit samples, the proposed method aims to detect harmful model updates efficiently while reducing the incentives for providers to manipulate audited models. Experiments show that Twister achieves higher detection power with fewer queries than uniform and performance-based monitoring, while substantially reducing manipulation gains under the considered adversarial setting.

Critical strengths
  1. The paper addresses the problem of maintaining audit validity after model deployment, which is an important research direction for ensuring the continuous trustworthiness of ML systems.
  2. The proposed Twister algorithm introduces a bi-level optimization formulation that explicitly considers the conflicting objectives of auditors and model providers.
  3. The paper is generally well-written.
Critical weaknesses
  1. The problem of continuously monitoring deployed models through black-box queries has been explored in previous studies, such as [1]. This work has already investigated statistical approaches for detecting behavioral changes or monitoring model fairness. Although the proposed Twister algorithm introduces bi-level optimization to account for an adaptive provider's utility-maximizing behavior, the paper does not sufficiently establish its advantages over existing monitoring approaches. The experimental evaluation primarily compares against uniform sampling and a standard performance-based test, without incorporating stronger monitoring or active query-selection baselines. The authors should provide a clearer discussion of the methodological differences from prior work and conduct more comprehensive comparisons to substantiate the claimed contributions.
  2. The experiments are restricted to two benchmark datasets, Folktables and CelebA, using a linear model and a simple CNN, respectively. These relatively simple architectures may not adequately represent the complexity of modern deployed ML systems. In particular, the proposed Twister algorithm relies on bi-level optimization to simulate the provider's response, raising concerns about its computational scalability to larger and more complex models. Although the paper reports computational costs for the evaluated models, these results do not establish how the method scales with model size or dataset size. The authors should consider evaluations on more modern architectures and provide additional scalability analysis to demonstrate the generalizability and practical applicability of the proposed framework.
  3. Although Twister demonstrates improved detection efficiency for harmful model updates, the results in Figure 2 indicate that the proposed method may also produce relatively high false alarm rates for beneficial updates. For example, in the Lending scenario, the average rejection probability reaches approximately 0.26 for updates that improve auditor utility by 0.13, with some model pairs exhibiting rejection probabilities above 0.6. This is concerning because the proposed framework triggers a new full white-box inspection whenever an alarm is raised, potentially introducing substantial unnecessary auditing costs. Moreover, although the framework is intended for periodic monitoring, the experiments primarily evaluate individual monitoring runs without investigating the cumulative false alarm probability or auditing overhead across repeated monitoring rounds. The authors should provide a more systematic analysis of the trade-off between detecting harmful updates and avoiding false alarms, particularly under long-term deployment scenarios.
  4. The manuscript contains several unfinished placeholders. For example, Appendix A (Extended Related Work) contains the placeholder text "Common techniques include blah bla blaw." . Additionally, Section 4 contains an unresolved citation placeholder, "CITE[celeba]", in the description of the CelebA dataset. The authors should carefully revise the manuscript and resolve these issues.

[1] Richter, Leo, et al. "An auditing test to detect behavioral shift in language models." International Conference on Learning Representations. Vol. 2025. 2025.

Questions for authors

See Weaknesses above.

3 comments

Dear Reviewer z97c, our common answers address your concerns about the number of scenarios and computational scale, the corrected Lending false-alarm results, and the unfinished placeholders.

Relation to change detection literature Our contribution is inspection-time query-distribution design that anticipates a utility-maximizing provider's response. [RHMK24] (which we also cite) is close to our work in that it aims at detecting changes that affect a score and not just any model change. However, motivated by a regulation scenario, we add an additional assumption: the knowledge of the provider training objective to inform which points to select.

Relation with active testing literature Similar to us, active testing aims at selecting the most "informative" queries to estimate or test some metric. However, active testing setups either assume full access to the model, which we don't assume after the audit or only black-box queries. Moreover, [GLMT+24] showed that in adversarial settings, no active testing strategy can perform better than uniformly random queries, impossibility that we bypass by adding a monitoring phase after the query selection.

Continuous monitoring We choose to expose our method (optimize the test distribution) on a simple binomial test to ease exposition. Twister can straightforwardly be extended to the testing-by-betting framework of [RHMK24].

References

[GLMT+24] Augustin Godinot, Erwan Le Merrer, Gilles Trédan, Camilla Penzo, and François Taïani. Under manipulations, are some AI models harder to audit? In 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 644–664, April 2024.

[RHMK24] Leo Richter, Xuanli He, Pasquale Minervini, and Matt Kusner. An auditing test to detect behavioral shift in language models. In The Thirteenth International Conference on Learning Representations, October 2024.

Thank you for the response.

While some concerns have been addressed, the scalability and long-term monitoring overhead remain insufficiently evaluated. Additionally, the presence of unfinished placeholders in the original submission raises concerns about the manuscript's readiness. Therefore, I will maintain my original score.

Dear Reviewer z97c,

Thank you for your fast response and for acknowledging some of your concerns were addressed. On the scalability concern you had, as we mention in the common response, we added a moderation scenario and demonstrated that our method can be applied to LLM scale. Is there a specific model/scenario you would like to see in the paper?

Best regards, Paper authors

Reviewer_oJLbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to solve a research challenge: when to remind auditors to conduct a fresh audit by monitoring model updates. They propose a framework named ∆-Audits. The paper first defines a generic monitoring mechanism (a disagreement test between the deployed and inspected models) and defines the role of the query distribution ρ. Then they propose ∆-Audits, which chooses ρ wisely by anticipating a utility-maximizing provider's response in a bi-level formulation, solved by Twister. In the experiments, the authors compare their method with uniform monitoring and an unpaired performance test on tabular and image datasets. The results show that Twister detects harmful updates with far fewer queries than uniform monitoring, but with the cost of more false alarms.

Critical strengths
  1. The research challenge is well-motivated and the authors give a formal formulation. Inspection followed by query-limited monitoring matches real audits, and the examples are concrete.
  2. The experiments show the proposed method's (Twister) strengths and also its weaknesses. E.g., the higher false-alarm rate on large beneficial updates in the Lending scenario.
Critical weaknesses
  1. The scenarios studied in the algorithm have limitations. For instance, the paper assumes that the provider knows the audit set D. If so, the provider can keep its predictions on D unchanged and alter the model elsewhere (such as what Meta did, "reportedly removed only the ads that surfaced in regulators' known queries"), and this will lead to bad results against ∆-Audits. Moreover, the auditor is also assumed to have the provider's full training data and pipeline, which is a strong assumption for the regulatory setting, for instance when auditing a proprietary model whose training data the auditor cannot obtain.

  2. The baselines in the experiments are simple, and the experiments only contain two datasets/scenarios. The baselines used are uniform ρ and an unpaired accuracy test. Another possible baseline is ρ proportional to the auditor's own weights, for instance.

  3. he paper seems incomplete. For instance, the related work in Appendix contains placeholder (line 705: 'Common techniques include blah bla blaw'). Overall, the readability and presentation can be improved.

Questions for authors
  1. Sec. 4 says the fingerprint hyperparameters are tuned "to minimize the manipulation gain on the auditor's data", and Figure 3 reports manipulation gain on that same data. Is this result biased by in-sample selection? Can the authors report it on a held-out audit set?

  2. False alarms on large benign updates are high in the Lending scenario, and each alarm triggers a costly inspection. How should ρ or gamma be chosen to improve this?

  3. How does ∆-Audits handle a provider that keeps its predictions on D unchanged and alters behavior on D? (See weakness 1.)

4 comments

Dear Reviewer oJLb, our common answers address your concerns about caching, inspection access, the number of scenarios, auditor-weighted baselines, the evaluation split (Q1), and the unfinished placeholders.

Q2: False alarms on large benign updates. This issue is now fixed: a bug changed the Lending data split within an experiment. We corrected it to keep the split fixed, as in the other scenarios; the updated Figure 2 shows rare false alarms on Lending, including for large benign updates.

Dear Reviewer oJLb,

Thank you again for reading and evaluating our work. Given our clarifications, new experiments and answers to your three questions (Q1 and Q3 in the common response), would you consider updating your overall rating? We’d be happy to clarify anything that remains open.

Best regards, Paper authors

Reviewer_oJLbOpenReview ↗

Thank you for your reply.

I thank the authors for their reply. However, there are still some questions that are not fully addressed. For instance, regarding the condition that the auditor needs to have the full access to the training data and pipeline. They argue that the model misspecification might hinder performance. This still limits the deployment of the auditor. Moreover, the concerns regarding the manipulation gain biased by in-sample selection: the authors argue the audit is defined on D by design. But this can still leads to results that are biased by data selection. At this stage, I would remain my original score.

Dear Reviewer oJLb,

Thank you for your answer and for engaging in the discussion. Here are further clarifications about access to training data and pipeline and the definition of the audit on D.

The objective of our work it to provide a method to check audit validity after a white-box audit even if the auditor looses access to the system. We do not tell the auditor how to select the evaluation data D, the auditor is responsible for mitigating data biases and other sample representativity issues. Again, the guarantee we provide and evaluate is "after a white-box audit on D, the auditor has a query-efficient monitoring scheme that allows to raise an alarm if the performance on D drop". We do not study classical generalization claims such as "after an audit on D, the model will work well on any D' close to D".

What we propose in this paper is to go beyond the black-box change detection literature and use the white-box access that regulators now have thanks to new EU regulations to improve post-audit monitoring. For compliance checks or investigations within their remit, EU regulators can obtain access to model weights and source code (AI Act Art. 92), high-risk systems’ training, validation and testing data (AI Act Art. 74), training-pipeline documentation (AI Act Arts. 53(1)), and relevant platform or gatekeeper algorithms, databases, testing information and data-handling practices (DSA Arts. 40, 69,72, and DMA Arts. 21 and 23).

Therefore, the access requirement determines when the method applies, and the choice of D determines how broadly the audit finding can be interpreted by the auditor. However, our contribution addresses what happens next: after a white-box audit on D, the auditor can detect a drop in performance on that same D with few queries, even after privileged access ends.

We hope this clarifies the scope of our guarantee and would be happy to discuss any remaining questions.

Reviewer_vbw6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that a one-time audit of an ML system goes stale once the provider updates the model. It proposes ∆-Audit, which pairs a white-box inspection of the model h0 with cheap black-box monitoring afterward. Monitoring is an exact one-sided binomial test of whether the deployed model hm disagrees with h0 on more than a fraction γ of audit points, where points are sampled from a weighting ρ.

The central idea is to choose ρ at inspection time by solving a Stackelberg (bi-level) problem. The auditor commits to ρ, a rational provider then maximizes its own utility subject to ρ-weighted disagreement staying below γ, and the auditor picks the ρ that minimizes its own utility loss under that best response. Twister solves a penalized, differentiable version of this problem using the single-loop solver.

Critical strengths
  • The problem is real and well framed. Post-audit drift is a genuine gap in audit regimes. Modeling the provider's response as a Stackelberg follower is the right abstraction, since the auditor commits first and the provider adapts.
  • The critique of uniform disagreement makes sense. The scam-ad example shows clearly that uniform disagreement fails in both directions: it is too lenient when a small but harmful slice changes, and too strict when beneficial changes are large.
Critical weaknesses
  • Section 3.1 assumes the provider knows D, h0, and ρ, and every monitoring query is drawn from D. A provider with enough capacity can serve h0's predictions on D and the unconstrained model everywhere else. That gives δρ = 0 for any ρ, so it passes every test while recovering the full manipulation gain on real traffic. This is exactly the Meta scam-ad behavior cited in the introduction (removing only the ads that surfaced in regulators' known queries), which the method cannot detect. The results look good only because manipulation gain and auditor cost are measured on D alone (Definition 2), and because the admissible class (a linear model, a small CNN fine-tuned by gradient descent) cannot special-case D. The authors should report both quantities on held-out data and include a provider that explicitly conditions on x ∈ D, or a high-capacity model.

  • In Proposition 1, κ(ρ) = maxᵢ wᵢᵃᵘᵈ/ρᵢ is minimized at ρ = wᵃᵘᵈ, which gives κ = 1. So the paper's own bound suggests sampling by auditor weight as the natural choice. An even simpler heuristic is ρ ∝ wᵃᵘᵈ·1[h0(xᵢ) = yᵢ]: flipping a correct prediction can only hurt the auditor, and flipping an incorrect one can only help. Neither baseline is evaluated, so the value of the bi-level machinery over a one-line heuristic is unproven. A paired labeled test is also missing, even though labels in D are available to the auditor.

  • "Harmful" is defined as any Uaud drop > 0, which includes drops within seed noise and drops that still satisfy τ. The unpaired test targets Uaud < τ, so scoring it on Λ > 0 judges it against a goal it was not designed for. Twister also has a higher false alarm rate in Lending, so part of its power advantage may come from being more trigger-happy. In addition, γ is set from uniform disagreement between seeds and then reused for the ρ-weighted test, without reporting the false alarm rate under pure seed changes for each test. The value γ = 0.2 (20% seed-to-seed disagreement for a linear model) also nearly guarantees uniform monitoring cannot fire, which inflates the "order of magnitude" claim. The paper should report ROC-style curves or power at a fixed empirical false positive rate, and sweep γ.

  • The fingerprint parameters are tuned "to minimize the manipulation gain on the auditor's data," and manipulation gain is then reported on that same D. This is selection on the test set, so the RQ2 results are optimistically biased. A held-out split for tuning is needed.

  • Twister simulates a gradient-based provider that starts from θ0 and has a known Uprov, and the evaluated manipulation uses the same model class, data, and utility. The paper never tests how well ρ holds up under a misspecified provider utility (for example, a different c_fp/c_fn), a different architecture or optimizer, retraining from scratch, or a different λ. Since the auditor will never know the provider's utility exactly, this is the central robustness test for the method. There is also a design/evaluation mismatch: ρ is designed against a penalized response but evaluated against a hard-constrained one, and the paper never reports whether the penalized responses actually satisfy δρ ≤ γ.

Questions for authors

Covered in weaknesses.

2 comments

Dear Reviewer vbw6, our common answers address your concerns about caching, inspection access, auditor-weighted baselines, the manipulation-gain evaluation split, and the corrected Lending false-alarm results.

The unpaired baseline tests utility lower-bound. Since a honest platform optimizes its own utility under auditor utility constraint τ\tau, we have Uaud(h0;D)=τU_{\text{aud}}(h_0; D) = \tau. Thus, testing Uaud(hm,D)≥τU_{\text{aud}}(h_m, D) \geq \tau and Uaud(hm,D)≥Uaud(h0,D)U_{\text{aud}}(h_m, D) \geq U_{\text{aud}}(h_0, D) are equivalent in our setup.

γ\gamma is tuned on uniform agreement We choose to tune γ\gamma with respect to uniform agreement to be able to compare uniform and weighted Twister.

We thank the reviewer for their suggestion for calibrating γ\gamma via a ROC curve. Such a calibration step would make sense if we had a classifier as the detector. A classifier would balances false positive and false negative rates and an ROC-curve would be right way to calibrate γ\gamma as a hyper-parameter. Instead in our design, we have a one-sided hypothesis test which only bounds false positives strictly. It is one-sided because to calibrate the test we cannot have label-balanced samples as input to the test. This is not a limitation of our method, but rather a consequence of a delta testing: it is unreasonable to assume the provider would ever updates the model to decrease performance. Nevertheless, we added a ROC curve analysis of our method and the corresponding baselines in Appendix F.

Twister optimizes a penalized relaxation of the true problem For practical reasons, we choose to relax the monitoring constraint into a penalization for the fingerprint optimization only. For the evaluation, we optimize the true manipulation under monitoring constraint problem. Thus our experiments can be seen as the minimum performance an auditor could reach; pending advances in constrained bi-level optimization.

Dear Reviewer vbw6,

Thank you again for taking the time to read and evaluate our work. Given our clarifications, new experiments and answers to your concerns, would you consider updating your overall rating? We’d be happy to clarify anything that remains open.

Best regards, Paper authors

Comments on the paper

1 comment

Dear reviewers, thank you very much for taking the time to read and evaluate our work. We are pleased that you all find the problem as interesting as we do and that you recognized the effort we spent in the formalization, write-up and experiments. In this comment we address points raised by multiple reviewers and will address individual comments under your respective reviews. We aimed for brevity, but we look forward to continuing the discussion and answer any remaining questions. The changes are colored in orange in the revised manuscript.

Caching & high capacity models (Rvbw6 and RoJLb) It is easy to escape monitoring if you can memorize all the labels on the audit set. This has been formally proved by [GLMT+24] in the case of a provider trying to escape the audit at all costs. In this work, assuming rational providers, we only consider manipulations that could increase the provider's utility. The provider can still escape monitoring, but at a cost to its utility, as studied in by the manipulation-gain experiments (section 4.2 and figure 3).

Small scale experiments (RoJLb and Rz97c) The experiments are on two datasets and small models, using a technique (bi-level optimization) whose scaling is uncertain. We added toxicity classification on about 20,000 CivilComments examples using a MiniLM encoder with rank-88 LoRA adapters. This extends the evaluation to a pretrained transformer, without establishing scalability to full-model fine-tuning. Moreover, recent developments in bilevel optimization have shown it can be applied to larger scale [PZZ+25]. Since our work is not tied to a specific bi-level algorithm, it will benefit from advances in the field.

Perfect knowledge during audit (Rvbw6 and RoJLb) Twister requires perfect knowledge to optimize the queries. Model misspecification might hinder performance. In our threat model, monitoring only assumes black-box label queries. Knowledge of training code, data, losses, etc comes from the inspection phase that precedes monitoring. It is not an assumption we make arbitrarily: the DSA, DMA and EU AI act all have mechanisms that allow regulators to ask about any part of the system, including the training data and objective. In cases where full white-box access is not yet possible, literature on the transferability of fingerprint/adversarial examples suggests that even an approximation of the true system could prove useful for Δ\Delta-Audits; we leave this for future work.

Auditor-weighted monitoring (Rvbw6 and RoJLb) Another baseline beyond uniform monitoring is to sample with probability proportional to auditor utility weights. We added both suggested query distributions: ρi∝wˉiaud\rho_i\propto\bar{w}^{\mathrm{aud}}_i and ρi∝wˉiaud1[h0(xi)=yiaud]\rho_i\propto\bar{w}^{\mathrm{aud}}_i\mathbf{1}[h_0(x_i)=y^{\text{aud}}_i], which restricts auditor weighting to initially correct predictions. Twister remains competitive against these baselines.

Optimizing the fingerprint on DD (Rvbw6 and RoJLb) The fingerprint is optimized on DD and you evaluate the manipulation gain on DD, isn't this selection on the test set? No. We corrected the text: manipulation gain is measured on the provider's dev set, as it already was in the submitted experiments. The audit quantities are defined on DD: the inspection requirement Uaud(h0,D)≥τU_{\text{aud}}(h_0, D) \geq \tau, the monitoring test, which only queries points of DD, and the auditor utility deterioration we seek to detect. We make no claim that monitoring protects the model's behavior outside DD. If the model was compliant on DD but then turned non-compliant on DD, the provider has to be held responsible. Moreover, it is necessary to separate the problem of detecting input distribution shifts and detecting model updates. Had the auditor used a different sample D′D' for monitoring, the company could have argued that the mismatch in measurements are due to difference in samples. Choosing a representative DD is a question every audit faces, one-shot or not; Δ\Delta-Audits answer a different one: once a model is certified on DD, has an update invalidated that certification? However, we do evaluate the generalization with respect to model changes. The model changes simulated by the bi-level fingerprint optimization are dropped once ρ\rho is returned and we re-train different model changes to evaluate whether Twister can detect changes that it did not see during optimization. We apologize for the unfinished placeholders in the submission; these and the missing CelebA citation have been corrected.

References

[GLMT+24] Augustin Godinot, Erwan Le Merrer, Gilles Trédan, Camilla Penzo, and François Taïani. Under manipulations, are some AI models harder to audit?

[PZZ+25] Rui Pan, Dylan Zhang, Hanning Zhang, Xingyuan Pan, Minrui Xu, Jipeng Zhang, Renjie Pi, Xiaoyu Wang, and Tong Zhang. ScaleBiO: Scalable bilevel optimization for LLM data reweighting, May 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.