acceptodds
Under review as a conference paper at ICLR 2027

PAIRAUDIT: Guiding Human Review with Graph Tokens under Distribution Shift

Abstract

Intrusion detectors can confidently misclassify attacks that were not seen during training. Human review can correct these errors, but only a limited number of cases can be checked. Uncertainty-based review may overlook confident errors, while anomaly scores alone do not show whether changing the review plan will correct more errors. We introduce PAIRAUDIT to find overlooked errors and improve review under a fixed budget. Its graph tokens capture prediction patterns across connected nodes. Rather than building another predictor through feature aggregation, PAIRAUDIT uses unusual relational patterns to uncover potential errors in existing predictions. Human feedback then helps decide whether these findings justify changing review priorities. Experiments across security tasks show that PAIRAUDIT corrects more errors on average than uncertainty-based review, including more errors on unseen attacks. These gains account for all review costs and do not require retraining the detector.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph learning, out-of-distribution detection, human review allocation, sequential auditing, intrusion detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KaquOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAIRAUDIT allocates a fixed review budget to correct a frozen classifier under distribution shift. It models node and neighborhood prediction tokens with class-conditional Gaussian mixtures whose weights depend on node features, then proposes replacements for part of an uncertainty-ranked queue. A sequential paired audit queries ordinary class labels and uses a working Bayesian predictive probability to choose the remaining reviews. The paper derives exact budget and net-correction identities that include both audited endpoints and displaced reviews, and distinguishes them from the heuristic decision posterior. Experiments on four primary security shifts, prevalence and multiclass extensions, and a harmful-candidate control report net correction gains over uncertainty review without retraining the detector.

Critical strengths
  1. The budget accounting is explicit and consequential. Both audited endpoints consume review capacity, displaced Raw reviews appear in the gain identity, and a favorable remaining exchange is distinguished from positive total workflow gain. This avoids overstating improvements by treating the audit as free (Sections 4-5).

  2. The statistical scope is carefully delimited. The finite-population sampling result does not require independent graph errors, while the Dirichlet predictive probability is correctly described as a working posterior rather than a distribution-free safety guarantee (Appendix C). This distinction makes the formal contribution easier to assess and use responsibly.

  3. The complete policy produces useful empirical gains on several frozen-detector tasks after paying all review costs, including low-OOD and multiclass settings. The paper also reports unfavorable cases: Trust Score gives more total corrections in two primary tasks, relational information is not uniformly helpful, and rejecting the Physics candidate still incurs a net loss. This measured evidence supports weak acceptance despite the attribution and audit-validation gaps below.

Critical weaknesses
  1. The contribution of the proposed graph score and adaptive audit is not fully separated from priority and retention choices. Table 4 is a valid equal-budget comparison of complete policies, but external scores do not receive the same priority/retention/audit wrapper. On UNSW, benign-priority Raw already gains 91.45 corrections versus PAIRAUDIT's 102.95 (Table 7); fixed mixture weights are close to the full method on several tasks, and node-marginal scoring is better on RDOS (Table 9). These controls are informative, but a matched-policy comparison and the same-candidate/no-audit control on the primary security tasks are needed to establish which proposed components add value.

  2. Decision reliability near the accept/reject boundary is under-tested. The primary tasks were selected retrospectively, all 21,600 primary audit runs authorize, and the main rejection example is from Coauthor Physics. Repeating pair permutations demonstrates stability for fixed workloads, not reliability across independent security shifts. Moreover, the stopping probability concerns the remaining paired exchange rather than total net gain or literal Raw continuation; Appendix E.2's node-marginal TON control shows that these targets can differ substantially. The paper states these limitations honestly, but the practical audit claim would be stronger with same-domain near-zero and harmful candidates and explicit calibration of realized workflow loss.

Questions for authors
  1. How do Trust Score or a simpler node-only score perform when given the same priority, Raw retention, and sequential-audit policy? Please also compare the proposed candidate with no audit and a fixed-size audit on the primary tasks at the same total review budget.

  2. Can the authors evaluate near-zero and negative candidate exchanges within the security tasks, preferably on prospectively held-out shifts, and report authorization probability, total net gain, and loss against literal Raw continuation? This would clarify when the paid audit improves the complete workflow rather than merely confirming an obviously favorable candidate.

  3. Can the frozen prediction banks and audit ledgers referenced by the reproducibility statement be made available anonymously? The source package explicitly omits them and notes that its portable preprocessing/backbone adapters do not exactly reproduce every historical producer; these artifacts would allow independent replay of the reported comparisons.

1 comment

Response to Q1

Thank you for suggesting these controls. We added comparisons on the four primary tasks: three reconstructed training fits per task, each with 600 audit orders. These are new reruns, not the original paper averages.

Entries: GG / GOODG_{OOD} / decision cost (% of HH).

Regarding “the same priority, Raw retention, and sequential-audit policy”:

Only the score changes. Node marginal removes explicit neighbor coordinates but retains the frozen GNN and gate.

TaskTrustNode marginalPairAudit
UNSW131.20 / 131.79 / 5.09%107.33 / 109.16 / 5.08%142.12 / 143.32 / 5.08%
CICIDS812.14 / 688.00 / 5.01%530.14 / 519.33 / 5.01%707.89 / 794.00 / 5.01%
RDOS-11.24 / 0.14 / 16.53%256.36 / 318.01 / 5.07%347.12 / 464.33 / 5.07%
TON62.24 / 38.22 / 5.08%-27.02 / -1.00 / 26.72%82.61 /103.91 / 6.77%

Overall, PairAudit outperforms Trust and Node marginal.

Regarding “no audit and a fixed size audit”:

We fix PairAudit's proposal. Fixed 10% decides once after about 10% of HH labels, using the same authorization and fallback rules. Both pair endpoints count. Sequential auditing is unchanged.

Entries: GG / GOODG_{OOD} / decision cost (% of HH).

TaskNo auditFixed 10%Sequential
UNSW144.00 / 145.33 / 0%140.25 / 141.67 / 10.04%142.12 / 143.32 / 5.08%
CICIDS737.00 / 821.67 / 0%685.00 / 769.67 / 10.03%707.89 / 794.00 / 5.01%
RDOS345.00 / 464.33 / 0%350.61 / 464.33 / 10.01%347.12 / 464.33 / 5.07%
TON91.00 / 113.67 / 0%73.95 / 93.40 / 10.17%82.61 / 103.91 / 6.77%
Physics-257.00 / −220.67/ 0%−47.04 / −46.15 / 10.14%-22.89 / -21.81 / 5.15%

Better no audit results on favorable candidates do not mean that auditing is unnecessary. We cannot know before review whether a candidate will help. On the four primary tasks, PairAudit achieves similar net gains, differing from direct adoption by at most 29.11 corrections, or 1.07% of HH, while making decisions with about 5% to 7% of HH on average.

Physics shows why this check matters. Direct adoption gives G=−261.67G=-261.67, whereas PairAudit rejects at the first look in all 1,800 runs and limits the mean loss to G=−22.89G=-22.89. It also preserves 24.16 more corrections than fixed 10% auditing, using roughly half the decision labels. Together, these results show fast decisions on favorable proposals and early loss control on harmful ones. All audit costs are included, and every policy still completes exactly HH reviews.

Response to Q2

Thank you for suggesting this test. Each method column reports mean GG / GOODG_{OOD} across three training fits. The final two columns report PairAudit's authorization rate and mean audit cost.

TaskTrust ScoreEnergyGNNSafeGRASPPairAuditPA authorizationPA audit cost / HH
BruteForce-186.33 / -3.00-18.33 / -2.33-122.33 / -3.00-162.00 / -3.00-4.38 / +0.060%5.53%
Dictionary-114.00 / +4.00-20.00 / -1.67-113.00 / -7.00-165.67 / -8.67-7.15 / +0.190%5.49%
Resources+27.67 / +73.67-18.00 / -12.00-49.33 / +37.67-220.33 / -115.33+228.59 / +336.63100%5.12%

These three tasks are unseen native attack subtype holdouts from X-IIoTID: BruteForce, Dictionary, and Discovering_resources (abbreviated as Resources). For each task, the named subtype is excluded from training and validation and treated as OOD during evaluation. PairAudit achieves higher mean total GG than all four external baselines on each task.

On BruteForce and Dictionary, PairAudit rejects every proposal with mean audit costs of only 5.53% and 5.49% of HH. Early rejection limits mean losses to 4.38 and 7.15 corrections. On Resources, PairAudit authorizes every run and achieves mean G=+228.59G=+228.59 and GOOD=+336.63G_{OOD}=+336.63 with an audit cost of 5.12% of HH. PairAudit can retain substantial gains from a useful proposal while limiting exposure to harmful proposals.

Response to Q3

Thank you for this suggestion. We will explore how best to share the experimental details and reproduction instructions with you while preserving anonymity during review. At the latest, we will publicly release the project at the camera-ready stage through accessible channels such as GitHub, including the source code, software usage instructions, experimental configurations, and reproduction procedures. Thank you for your understanding and patience.

We are especially grateful for the depth, precision, and constructive spirit of your review. Your questions reflect a careful understanding of our framework and the practical challenges of budgeted human review. They have guided us toward more informative experiments and a clearer presentation of our contribution.

Reviewer_G2o2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A detector can miss unfamiliar attacks with high confidence, leaving these errors outside an uncertainty-based review queue. PAIRAUDIT uses predictions from each node and its neighbors to find possible replacements, then reviews sampled pairs to decide whether to change part of the queue. Both members of each audited pair count toward the review budget, and the detector remains fixed. Across four primary tasks, the method corrects an average of 80.01 to 789.79 more errors than uncertainty-based review after accounting for audit costs.

Critical strengths
  1. Equation 4 and Proposition 1 account for both audit labels and the original reviews they displace, making the reported gains meaningful under a fixed budget.
  2. The gains on all four primary tasks, together with the low-OOD and multiclass tests, provide useful evidence that the policy can recover errors missed by uncertainty-based review.
Critical weaknesses
  1. Table 4 changes both the ranking and the review policy, while the node-marginal model beats the full model on XIIoTID in Table 9, leaving the added value of graph tokens unclear.
  2. All four primary tasks authorize every exchange, while the rejection control uses a clearly harmful proposal, so the experiments say little about whether the paid audit helps when replacement gains are small.
Questions for authors
  1. How does Trust Score perform with the same class priority, queue retention, and paid audit as PAIRAUDIT?
  2. What gains result from accepting the same proposed exchanges without an audit on the four primary tasks?
  3. How often does authorization lead to a negative total gain when the expected gain from the remaining exchange is close to zero?
  4. Do the results hold on additional attack subtypes selected independently of the reported experiments?
Reviewer_WiouOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAIRAUDIT allocates a fixed human-review budget to correct errors made by a frozen classifier under distribution shift. It scores prediction patterns using a feature-conditioned density over node and neighborhood posteriors, then proposes replacements for part of an uncertainty-based queue. A sequential pair audit uses ordinary class labels to decide whether to adopt the remaining replacements. Both endpoints of each audited pair count toward the budget. The reported experiments show positive mean overall and unseen-attack correction gains on four primary security tasks, with additional prevalence, multiclass, and rejection controls.

Critical strengths

1.The paper evaluates errors corrected within a fixed budget and charges both labels in every audited pair. Proposition 1 also makes the opportunity cost explicit: an audit can find errors while displacing other useful reviews. The supplied implementation retains all queried endpoints and constructs exactly H reviews; the corresponding tests passed. This makes the reported gains easier to interpret and is a meaningful contribution to evaluating human-review policies. 2.The four primary tasks show mean net gains of 102.95, 789.79, 340.30, and 80.01 corrections over Raw after audit costs. The low-prevalence and multiclass results extend the evidence beyond the primary setup. TON also provides a concrete case where explicit neighborhood information helps: removing the neighborhood coordinates changes the gain from +80.01 to -39.86. These results justify taking the approach seriously, even though the component benefits vary across tasks.

Critical weaknesses

1.Table 4 compares complete policies, but PAIRAUDIT combines a new score, class priority, Raw retention, and paid auditing, while the external methods use their rankings directly. This leaves the value of the proposed components unresolved. Table 9 makes the issue consequential: feature conditioning adds only 1.26, 1.68, and 1.85 mean corrections on CICIDS, X-IIoTID, and TON; the node-marginal control outperforms the full score on X-IIoTID. The benign-priority Raw control also already obtains +91.45 on UNSW, versus +102.95 for PAIRAUDIT. I would like to see priority-matched rankings under the same retention and audit policy, together with an unaudited B∪C control on the primary tasks. Without these comparisons, I cannot judge how much of the improvement requires the proposed conditional graph score and audit. 2.The audit authorizes when the working posterior assigns at least 0.95 probability to a positive remaining pair sum. The stated objective is positive total gain G after paying for the audit. Proposition 1 shows why these events differ, and the paper acknowledges the distinction. All primary runs authorize, while all Physics runs reject, so these experiments mostly cover clear decisions. The supplementary docs/METHOD.md also states that the thresholds were chosen following retrospective sensitivity analysis, without independent validation of that choice. I would like to know which workloads informed this selection and see validation near zero exchange gain, including remaining-gain calibration, total-loss frequency and severity, and comparison with a simple fixed-audit rule under the same budget. This would establish the sequential rule's value in difficult decisions.

Questions for authors

1.Under identical class priorities, retained fractions, and audit schedules, how does the proposed score compare with Trust Score and priority-adjusted Raw? What is the gain of directly reviewing B∪C without an audit on each primary task? These comparisons would change my assessment of which parts of PAIRAUDIT are necessary. 2.Why is authorization based on the remaining pair sum rather than total net gain? Can the existing ledgers show, by stopping look, the relationship between p_m, the realized remaining gain, and G, including the size of losses? I am asking about the behavior of the current rule, rather than requiring a distribution-free no-loss guarantee. 3.Section 6.1 says the tasks were selected retrospectively, and docs/METHOD.md reports retrospective threshold selection. Which workloads and outcomes informed these choices, and was any evaluation set kept separate from threshold selection? Please distinguish variability across frozen fits from variability across audit orders and provide paired differences for the main comparisons. The 600 audit randomizations do not establish generalization to new attack events.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.