acceptodds
Under review as a conference paper at ICLR 2027

DynaAudit: Auditing Hidden Supervision-Channel Drift

Abstract

Weak-supervision systems are often monitored through features and source outputs, yet the mapping from those observables to the latent label can change after deployment. We study hidden supervision-channel drift, where distinct supervision mechanisms induce the same declared unlabeled monitor transcript while implying different gold-label posteriors. This creates a fundamental blind spot: under exact transcript equivalence, purely unlabeled detection has power no greater than its false-alarm rate, and unlabeled estimation incurs irreducible error. We characterize support auditability—whether trusted labels available on the deployment support can distinguish candidate mechanisms—and, for a finite correctly specified candidate family, derive an anytime finite-sample certificate for fully adaptive auditing based on accumulated pairwise Bhattacharyya information. The guarantee is acquisition-policy agnostic, separating statistical validity from acquisition speed. These results motivate DynaAudit, which separates representative sentinel detection, mechanism-aware diagnosis, model criticism and abstention, and inclusion-aware reuse of adaptively selected labels. Controlled and temporal experiments show that targeted audits concentrate diagnostic information when post-alarm labels are scarce, while matched-budget random auditing can catch up as total trusted-label budgets grow. Natural temporal data further show that prediction can improve even when no named mechanism is defensibly supported. Overall, DynaAudit characterizes when unlabeled monitoring is fundamentally insufficient and what additional trusted evidence is required for auditable adaptation.

Primary area:
learning theory
Keywords:
weak supervision, supervision drift, distribution shift, active auditing, sequential testing, adaptive data acquisition

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bTTrOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to detect changes in the reliability of weak labeling sources when their observed outputs may still look unchanged. The paper first shows that such hidden changes can be impossible to detect from unlabeled monitoring alone. It then proposes DynaAudit, which uses a small number of trusted labels for drift detection and mechanism diagnosis. The framework combines representative sentinel labels with targeted audits and adaptively chooses which samples to inspect. Experiments on synthetic and temporal datasets are used to evaluate the proposed auditing framework.

Critical strengths
  1. This paper identifies a meaningful blind spot in weak supervision: stable weak-source outputs do not necessarily mean that their relationship with the true labels is also stable.
Critical weaknesses
  1. The presentation of the paper is terrible. The main text is quite compressed with only 8 pages and figures are crowded and difficult to read.
  2. Many of the theoretical tools are based on existing results, so the novelty only comes from the problem formulation.
Questions for authors
  1. How does DynaAudit behave when the true drift mechanism is not included in the predefined candidate family?
  2. Why does mechanism diagnosis remain inconclusive in many cases on the natural temporal datasets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bTTr

If the true channel is missing, DynaAudit can name the wrong explanation. Theorem 3 gives no protection in that case. We therefore report a separate check of the selected candidate using 100 additional labels. At the strongest tested departure, after SR-MKL diagnosis, this check rejected in 99 of 100 IMDb trials, but in only 1 of 100 SMS trials and 6 of 100 YouTube trials. The same perturbation produced different distances from the candidate family on the three datasets. In particular, a pass on SMS is not persuasive evidence that the selected explanation is right (Section 8; Appendix H).

The natural gas result has a different explanation. Its 78.72% mean “unknown” rate came from a fit threshold: even the best candidate’s normalized loss exceeded the historical cutoff. It was not a failure to cross the sequential stopping boundary. We checked whether the size of the candidate family explained the result. The flag remained frequent with four grouped candidates (79.62%) and with two (82.48%). That tells us that simplifying the family did not fix the problem. Without independent mechanism labels, we cannot attribute individual flags to misspecification rather than limited evidence.

We agree that the sequential-testing machinery is not new. What it lets us establish here is which diagnoses a particular audit can afford. In Table 3, four of five candidates cannot meet the Bhattacharyya threshold within 200 labels, however the queries are allocated. Figure 1 also shows where the standard likelihood-ratio rule is much cheaper. We think that budget distinction is useful without claiming a better general test.

I appreciate the authors' response and clarifications. However, I still have concerns about the limited methodological novelty and practical effectiveness of the proposed framework. In addition, the paper's writing and presentation could be further improved. Therefore, I will maintain my initial rating.

Reviewer_k9LaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to study “hidden supervision-channel drift, where distinct supervision mechanisms induce the same declared unlabeled monitor transcript while implying different gold-label posteriors.” However, the authors never define the meaning of the notation and the problem itself. In particular, how mechanism is formalized, how channel is defined, what are gold-labeled posteriors. Formal definitions, in lines 98-101 and 123-136 create even more ambiguity, for instance: is t time? What is XtX_t and Λt\Lambda_t? what is the meaning of the law of (full transcript) Z under mechanism θ\theta? What is the target TT of the mechanism θ\theta?

Without clarifying the problem and notation, it is impossible to understand the formulas in the Theorem. They introduce new notation without explaining them, for instance, in Theorem 1 there is d(,)d(,) which may be a kind of distance/metric (but if so, please confirm and state if there are any additional assumptions on the considered metric space), what is T^\hat{T} or TVTV. The proof is deferred to Appendix A, which contains several smaller statements without reference to Theorem 1 (I suspect Theorem 5 and the following Corollary 7 are equivalent to Theorem 1, but it is not clarified in the paper). Similar problem occurs in Theorem 2, and moreover, the reader is referenced to Appendix A for detail arguments while they are in fact in Appendix B.

Described contribution on page 2 provides even more notation, without explanation and connection to specific results. Theoretical results claimed in Theorems 1 and 2 are not compared against experimental results.

In the current form, I am unable to evaluate the submission, and therefore I recommend rejecting it.

Critical strengths

None

Critical weaknesses

Lack of sufficient formalization to understand theoretical claims. Experimental results do not refer to and do not complement/support theoretical results.

Questions for authors

The paper should be re-written entirely, to provide clear formalization and links between theory and experimental evaluation, se examples of ambiguity in the paper summary.

2 comments
AuthorsOpenReview ↗

Response to Reviewer k9La

The original presentation asked the reader to interpret the bound before we had said clearly what was being estimated. Table 1 now gives a concrete reading of Theorem 1. The two worlds produce exactly the same distribution of rule votes, but when both rules vote 1, the probability that the trusted label is 1 is 16/17 in one world and 27/34 in the other. That probability is the target. Their difference is 5/34, so an estimator using only the votes has worst-case expected absolute error of at least 5/68 in this example.

Section 2 defines a mechanism as a model for the joint stream and its transcript law as the distribution of the entire sequence the monitor sees. The general theorem uses a measurable metric on the target space; the example uses absolute distance. We also distinguish the item-arrival index from the number of labels queried. The estimation and detection arguments for Theorem 1 are Theorem 6 and Corollary 8 in Appendix A.2. Proposition 2 is expanded in Appendix A.3. The adaptive result, now Theorem 3, has its proof in Section 5 (pp. 4–5); its extensions are in Appendix B.

The link to the experiments is a prediction that could be checked against the runs. Section 4 shows that the two-world Bhattacharyya rule needs 117 audited agreements to assert at the 5% level. Under greedy acquisition it had asserted by 100 labels in 0 of 10,000 runs; the likelihood-ratio rule had asserted in 9,886 on the same trajectories. Those are stopping counts, not accuracy scores. They show why the original fixed-budget experiments could not, on their own, tell us what certification would cost.

Thank you, I appreciate your answer and editorial work. The presentation is slightly improved, but still not enough to understand technical nuances. I would start from describing the system architecture and its processes, while keeping consistent and formal definitions. The formalism cannot be avoided, as it occurs later in the technical analysis - let it occur in preliminaries, with detail description. For instance, I'd start from describing system components (sources, stream of items, ...), their capabilities/format, input, output, etc.

Reviewer_BKhjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors present DynaAudit for auditing hidden supervision-channel drift. The procedure includes a label acquisition strategy and an anytime-valid sequential test for detecting the drift, assuming the new ("shifted") model after the drift belongs to a set of predefined candidates. Improved identification of shifts are observed in experiments.

Critical strengths

The main auditing algorithm seems correct (at least the martingale constructions and statistical validity) under their assumptions

Critical weaknesses
  • Limited real world application: The main guarantee assumes a correctly specified family of candidate label distributions, but the paper does not demonstrate how to construct such a family in practice. Furthermore, everything holds only if the shifted model is known and constant during the auditing procedure.

  • Poorly written: I found the paper extremely difficult to follow, despite being familiar with sequential testing and machine learning. The introduction fails to explain the general research area and uses terminology that is only defined later. It'd be nice to add a motivating application beyond the toy example that only illustrates identifiability problem as fas as I understand. The contributions also need clearer positioning: I found it difficult to distinguish what is known from previous work from the main novelties of this paper. I had to piece together many basic points—what is being tested, what the candidate models represent, and which distributions are assumed known—because the notation, assumptions, and problem setup are not introduced coherently. Finally, the transitions between sections are abrupt: I often had to rely on the section titles to understand what was coming next, without a clear explanation of how the different parts fit into the overall argument.

Questions for authors

Please improve presentation and provide concrete real applications where the assumptions for testing make sense.

1 comment
AuthorsOpenReview ↗

Response to Reviewer BKhj

The practical limitation you identify is real: historical labels can suggest a family of channels, but fitting that family does not establish that the next change belongs to it. What the auditor knows in Theorem 3 is each candidate’s prediction for the next queried label, not which candidate is active. The active index must stay fixed during the audit. Its predictions may depend on the observed history; an unmodeled switch halfway through is not covered.

Section 3 now gives a construction one can reproduce. On WRENCH, we fit a reference model to historical labels and rule votes, construct alternatives that change the intercept, individual rule effects, or interactions, and freeze those alternatives before querying. The vote patterns come from real messages and reviews; the changed labels are simulated. A controlled replay therefore gives us a setting where the assumption holds by design, not evidence that it holds in a live deployment.

The spam example in the introduction gives this distinction a practical consequence. If two rules flag the same phrase, a team might spend its review budget on messages where they disagree. In our two-world example those messages are the hardest to classify, yet their labels tell us nothing about which world we are in. Only agreements distinguish the worlds. That choice of what to inspect is the contribution we want to convey; the sequential-testing tools themselves are established.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

The clearest way to see what changed is Table 3 (p. 8). In the five-candidate example, we can calculate before seeing a single trusted label which candidates the Bhattacharyya rule could name within a budget of 200. Only candidate 0 can meet the threshold. That does not make it the right answer: under greedy acquisition, the rule named 0 in 4,998 of 5,000 runs when 0 was true, but also in 11 of 5,000 when 1 was true.

This distinction between a possible assertion and a correct one now organizes the paper. Sections 2–5 state what the monitor observes, what the auditor must know, and what the stopping rule guarantees. Figure 1 reports label cost and wrong assertions alongside a standard likelihood-ratio rule, which usually stops sooner in these experiments. We identify the stopping-rule study as a post-review addition and keep its results separate from the original fixed-budget experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.