- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to detect changes in the reliability of weak labeling sources when their observed outputs may still look unchanged. The paper first shows that such hidden changes can be impossible to detect from unlabeled monitoring alone. It then proposes DynaAudit, which uses a small number of trusted labels for drift detection and mechanism diagnosis. The framework combines representative sentinel labels with targeted audits and adaptively chooses which samples to inspect. Experiments on synthetic and temporal datasets are used to evaluate the proposed auditing framework.
Critical strengths
- This paper identifies a meaningful blind spot in weak supervision: stable weak-source outputs do not necessarily mean that their relationship with the true labels is also stable.
Critical weaknesses
- The presentation of the paper is terrible. The main text is quite compressed with only 8 pages and figures are crowded and difficult to read.
- Many of the theoretical tools are based on existing results, so the novelty only comes from the problem formulation.
Questions for authors
- How does DynaAudit behave when the true drift mechanism is not included in the predefined candidate family?
- Why does mechanism diagnosis remain inconclusive in many cases on the natural temporal datasets?
2 comments
I appreciate the authors' response and clarifications. However, I still have concerns about the limited methodological novelty and practical effectiveness of the proposed framework. In addition, the paper's writing and presentation could be further improved. Therefore, I will maintain my initial rating.
Response to Reviewer bTTr
If the true channel is missing, DynaAudit can name the wrong explanation. Theorem 3 gives no protection in that case. We therefore report a separate check of the selected candidate using 100 additional labels. At the strongest tested departure, after SR-MKL diagnosis, this check rejected in 99 of 100 IMDb trials, but in only 1 of 100 SMS trials and 6 of 100 YouTube trials. The same perturbation produced different distances from the candidate family on the three datasets. In particular, a pass on SMS is not persuasive evidence that the selected explanation is right (Section 8; Appendix H).
The natural gas result has a different explanation. Its 78.72% mean “unknown” rate came from a fit threshold: even the best candidate’s normalized loss exceeded the historical cutoff. It was not a failure to cross the sequential stopping boundary. We checked whether the size of the candidate family explained the result. The flag remained frequent with four grouped candidates (79.62%) and with two (82.48%). That tells us that simplifying the family did not fix the problem. Without independent mechanism labels, we cannot attribute individual flags to misspecification rather than limited evidence.
We agree that the sequential-testing machinery is not new. What it lets us establish here is which diagnoses a particular audit can afford. In Table 3, four of five candidates cannot meet the Bhattacharyya threshold within 200 labels, however the queries are allocated. Figure 1 also shows where the standard likelihood-ratio rule is much cheaper. We think that budget distinction is useful without claiming a better general test.