- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PAIRAUDIT allocates a fixed review budget to correct a frozen classifier under distribution shift. It models node and neighborhood prediction tokens with class-conditional Gaussian mixtures whose weights depend on node features, then proposes replacements for part of an uncertainty-ranked queue. A sequential paired audit queries ordinary class labels and uses a working Bayesian predictive probability to choose the remaining reviews. The paper derives exact budget and net-correction identities that include both audited endpoints and displaced reviews, and distinguishes them from the heuristic decision posterior. Experiments on four primary security shifts, prevalence and multiclass extensions, and a harmful-candidate control report net correction gains over uncertainty review without retraining the detector.
Critical strengths
-
The budget accounting is explicit and consequential. Both audited endpoints consume review capacity, displaced Raw reviews appear in the gain identity, and a favorable remaining exchange is distinguished from positive total workflow gain. This avoids overstating improvements by treating the audit as free (Sections 4-5).
-
The statistical scope is carefully delimited. The finite-population sampling result does not require independent graph errors, while the Dirichlet predictive probability is correctly described as a working posterior rather than a distribution-free safety guarantee (Appendix C). This distinction makes the formal contribution easier to assess and use responsibly.
-
The complete policy produces useful empirical gains on several frozen-detector tasks after paying all review costs, including low-OOD and multiclass settings. The paper also reports unfavorable cases: Trust Score gives more total corrections in two primary tasks, relational information is not uniformly helpful, and rejecting the Physics candidate still incurs a net loss. This measured evidence supports weak acceptance despite the attribution and audit-validation gaps below.
Critical weaknesses
-
The contribution of the proposed graph score and adaptive audit is not fully separated from priority and retention choices. Table 4 is a valid equal-budget comparison of complete policies, but external scores do not receive the same priority/retention/audit wrapper. On UNSW, benign-priority Raw already gains 91.45 corrections versus PAIRAUDIT's 102.95 (Table 7); fixed mixture weights are close to the full method on several tasks, and node-marginal scoring is better on RDOS (Table 9). These controls are informative, but a matched-policy comparison and the same-candidate/no-audit control on the primary security tasks are needed to establish which proposed components add value.
-
Decision reliability near the accept/reject boundary is under-tested. The primary tasks were selected retrospectively, all 21,600 primary audit runs authorize, and the main rejection example is from Coauthor Physics. Repeating pair permutations demonstrates stability for fixed workloads, not reliability across independent security shifts. Moreover, the stopping probability concerns the remaining paired exchange rather than total net gain or literal Raw continuation; Appendix E.2's node-marginal TON control shows that these targets can differ substantially. The paper states these limitations honestly, but the practical audit claim would be stronger with same-domain near-zero and harmful candidates and explicit calibration of realized workflow loss.
Questions for authors
-
How do Trust Score or a simpler node-only score perform when given the same priority, Raw retention, and sequential-audit policy? Please also compare the proposed candidate with no audit and a fixed-size audit on the primary tasks at the same total review budget.
-
Can the authors evaluate near-zero and negative candidate exchanges within the security tasks, preferably on prospectively held-out shifts, and report authorization probability, total net gain, and loss against literal Raw continuation? This would clarify when the paid audit improves the complete workflow rather than merely confirming an obviously favorable candidate.
-
Can the frozen prediction banks and audit ledgers referenced by the reproducibility statement be made available anonymously? The source package explicitly omits them and notes that its portable preprocessing/backbone adapters do not exactly reproduce every historical producer; these artifacts would allow independent replay of the reported comparisons.
Response to Q1
Thank you for suggesting these controls. We added comparisons on the four primary tasks: three reconstructed training fits per task, each with 600 audit orders. These are new reruns, not the original paper averages.
Entries: G / GOOD / decision cost (% of H).
Regarding “the same priority, Raw retention, and sequential-audit policy”:
Only the score changes. Node marginal removes explicit neighbor coordinates but retains the frozen GNN and gate.
Overall, PairAudit outperforms Trust and Node marginal.
Regarding “no audit and a fixed size audit”:
We fix PairAudit's proposal. Fixed 10% decides once after about 10% of H labels, using the same authorization and fallback rules. Both pair endpoints count. Sequential auditing is unchanged.
Entries: G / GOOD / decision cost (% of H).
Better no audit results on favorable candidates do not mean that auditing is unnecessary. We cannot know before review whether a candidate will help. On the four primary tasks, PairAudit achieves similar net gains, differing from direct adoption by at most 29.11 corrections, or 1.07% of H, while making decisions with about 5% to 7% of H on average.
Physics shows why this check matters. Direct adoption gives G=−261.67, whereas PairAudit rejects at the first look in all 1,800 runs and limits the mean loss to G=−22.89. It also preserves 24.16 more corrections than fixed 10% auditing, using roughly half the decision labels. Together, these results show fast decisions on favorable proposals and early loss control on harmful ones. All audit costs are included, and every policy still completes exactly H reviews.
Response to Q2
Thank you for suggesting this test. Each method column reports mean G / GOOD across three training fits. The final two columns report PairAudit's authorization rate and mean audit cost.
These three tasks are unseen native attack subtype holdouts from X-IIoTID: BruteForce, Dictionary, and Discovering_resources (abbreviated as Resources). For each task, the named subtype is excluded from training and validation and treated as OOD during evaluation. PairAudit achieves higher mean total G than all four external baselines on each task.
On BruteForce and Dictionary, PairAudit rejects every proposal with mean audit costs of only 5.53% and 5.49% of H. Early rejection limits mean losses to 4.38 and 7.15 corrections. On Resources, PairAudit authorizes every run and achieves mean G=+228.59 and GOOD=+336.63 with an audit cost of 5.12% of H. PairAudit can retain substantial gains from a useful proposal while limiting exposure to harmful proposals.
Response to Q3
Thank you for this suggestion. We will explore how best to share the experimental details and reproduction instructions with you while preserving anonymity during review. At the latest, we will publicly release the project at the camera-ready stage through accessible channels such as GitHub, including the source code, software usage instructions, experimental configurations, and reproduction procedures. Thank you for your understanding and patience.
We are especially grateful for the depth, precision, and constructive spirit of your review. Your questions reflect a careful understanding of our framework and the practical challenges of budgeted human review. They have guided us toward more informative experiments and a clearer presentation of our contribution.