acceptodds
Under review as a conference paper at ICLR 2027

Auditing Model Fusion under Asymmetric Loss

Abstract

Systems that fuse several models under an asymmetric operational loss usually summarise each model by a scalar: a hard vote, or a reliability weight in place of a likelihood. We ask three questions about that practice, and answer each with a finite-sample object. Is it the scalar that hurts? A matched control fuses hard labels through one code path and varies only how each model is described: its fitted confusion matrix, against the symmetric channel carrying that matrix's own accuracy. This separates error structure from the change of representation that confounded earlier comparisons. The scalar description loses on all seeds of the primary domain, by a paired risk-weighted loss; the secondary domains show smaller or more sample-sensitive effects. Can the evaluation tell? An evaluation can be quantised more coarsely than the effect it measures. For a separate, smaller contrast against weighted averaging, we distinguish observed ranking sensitivity from uncertainty using an exact leave-one-out diagnostic— tighter than the bound it replaces—and a retrospective empirical-Bernstein calculation. What can a deployed rule certify? We give a one-sided finite-sample certificate that controls, at level , the probability of falsely declaring a deployed rule worse than a fixed comparator, built on where in the loss matrix the rule sends the rare class, and costing only the calibration split rather than the rare rows alone. It fires on all seeds of the decisive domain under the weaker of two concentration bounds. Where it cannot fire it abstains, and we report the abstention.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
model fusion, asymmetric loss, confusion matrix, Bayesian decision theory, rare events, intrusion detection, calibration, finite-sample audit

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_35bpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at auditing model-fusion rules when action losses are asymmetric. 3 contributions as i read it: (1) a controlled comparision of two hard-label fusion rules that share the same posterior-product calc, one uses a symmetric channel from each model's overall accuracy, the other uses the fitted class-conditional confusion matrix; (2) an evaluation audit - exact leave-one-out ranking sensitivity + a retrospective empirical-Bernstein resolution calculation; (3) a one sided finite-sample certificate that rejects the deployed rule if a lower bound on its rare-class risk contribution is above an upper bound on a fixed comparator's total risk.

Experiments on NSL-KDD, AI4I and CICIDS2017, 10 seeds. Clearest result is NSL-KDD: risk-weighted loss goes 112.295 -> 5.935,and the confusion-matrix arm wins on all 10 seeds. The other two domains show smaller / more sample sensitve gaps. Hoeffding certificate rejects on all NSL-KDD seeds and abstians on the other two. Authors are upfront that the naive Bayes combiner and the concentration inequalities are standard tools.

Critical strengths
  • The controlled comparison in Sec 4 is nicely done. representation, fusion formula, calibration split, prior, smoothing all fixed, only the channel description changes (scalar acc vs full confusion matrix). This removes the representation confound from the earlier comparison. The NSL-KDD gain is large and not driven by a handful of rows - Table 6 shows 284-316 of 500 rows get different actions per seed, max LOO sensitivity 0.0214. So I'm fairly convinced the symmetric channel throws away useful info in this setting.
  • I liked that the audit keeps observed sensitivity and statistical guarantees apart. The LOO stat says directly whether deleting one row flips the comparison; the certificate separates a plug-in comparison from a confidence-qualified rejection, and abstention is treated as inconclusive (good). App. G also clearly separates the calibration-fitted rules from the fixed rules on the certificate path. Useful for practitioners dealing w/ rare, costly errors where aggregate numbers can mislead.
Critical weaknesses
  1. Not clear what the rejection certificate adds over a simple, stronger baseline. On the certificate sample let W_i = L(A_i,Y_i) and X_i = 1{Y_i=y*} L(A_i,Y_i). Losses are nonneg so 0 <= X_i <= W_i <= L_max. If you plug the (observable) full deployed loss W_i into Thm 4 instead of X_i, the Hoeffding penalty is unchanged and the rejection margin is weakly larger on every sample - no extra observations or model evals needed, at least in the reported setup. A direct bound on the paired full-risk difference could do even better. The theorem is fine under its assumptions, my concern is more about its usefulness as a rejection procedure. Maybe the intended point is the narrower diagnosis "rare-class losses alone are enough to show inferiority"? If so pls state that objective explicitly and show why it matters vs direct total-risk comparisons. Since the paper positions the estimand (not the inequality) as the contribution i think this needs addressing.

  2. Relation to the existing fusion hierarchy. Kuncheva & Rodriguez (A weighted voting framework for classifiers ensembles, KAIS 2014) already have a common hard-label probabilistic framework going model accuracy -> class-specific recall -> full confusion matrix. Their setting is classification error, and extending to asymmetric action loss is relevant, but their hierarchy is a natural reference point and gives an obvious intermediate baseline. As Sec 4 itself notes, the current intervention relaxes two things at once (equal classwise diagonal accuracy, and uniform off-diagonal errors), so we cant tell whether class-specific recall already explains most of teh gain or whether detailed error destinations really add a lot. This matters for picking a fusion model when rare-class calibration data is limited. Also the headline effect is vs this specific symmetric arm; Table 4 has ~7.45 vs 5.93 against weighted averaging on NSL-KDD, much smaller. A focused look at the recall-preserving intermediate channel would strengthen the paper quite a bit imo, and clarify the scope of the practical conclusions.

Questions for authors
  1. Is there some observation restriction or a distinct auditing objective that makes the rare-class contribution preferable to the full deployed loss? Could you compare against direct full-risk and paired-risk certificates on the same Table 2 samples at matched confidence, and say what extra conclusion the rare-class version is meant to give?
  2. Could you add a (clearly labeled secondary) analysis with a channel that keeps Q_k(y,y) per class and sets each off-diagonal entry in row y to (1-Q_k(y,y))/(C-1)? Whether this recovers most of the NSL-KDD improvement would materially change my view of the new empirical insight beyond the accuracy/recall/full-confusion hierarchy.
  3. Sec 2 says the cached uncertainty-audit replay reproduces only 8/10 AI4I seeds, but the bootstrap widths are used together with the frozen-run point estimates. What justifies using those widths for the 2 non-reproduced seeds? Does an exactly reproduced bootstrap path give the same action-sensitivity results, and support the Sec 5 conclusion that eval uncertainty dominates calibration-fitting uncertainty? If this is already in S2 pls point me to it.
Reviewer_KkZEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies classifier fusion of five classical tabular models across three datasets (NSL-KDD, AI4I, and CICIDS2017) under a heavily asymmetric 4×44 \times 4 cost matrix where missing a rare FAILURE instance incurs a high penalty (10001000 vs. 1010 for false escalation). Building on prior work, the authors conduct three finite-sample analyses: (1) a matched ablation on hard labels showing that full confusion-matrix Naive Bayes combination outperforms a scalar-accuracy symmetric channel on NSL-KDD (while producing small or noisy differences on the other two datasets); (2) a leave-one-out sensitivity metric (ρLOO\rho_{\text{LOO}}) and empirical-Bernstein sample-size calculation demonstrating that the ntest=500n_{\text{test}} = 500 evaluation splits on AI4I and CICIDS2017 contain too few rare-class instances to reliably resolve method differences; and (3) a one-sided Hoeffding/empirical-Bernstein concentration bound on the calibration split to certify when a deployed rule performs worse than a fixed baseline.

Critical strengths

None

Critical weaknesses

In the second paragraph, the authors write "Giving the fuser each model’s full class-conditional error structure removes most of it—established in earlier work on this system, which left two questions open that this paper is about." with no reference to what this earlier work may be.

In short, the paper is unreadable to me. The introduction alone is littered with numbers that come out of nowhere and all the relevant definitions to even understand what is being studied are seemingly deferred to the appendix.

Stripping away the presentation, the authors explicitly note "Arm B is not a new rule and we claim nothing for it" when describing the main method of combining classifers using their held-out confusion matrices. Moreover, while machine-checking proofs is fine and useful, the mathematical statements themselves are elementary. Theorem 6 is a basic linear algebra identity, and Proposition 1 simply states Bernstein half-width is decreasing in nn.

Questions for authors

I have no questions and welcome a rebuttal from the authors in the event that I grossly misunderstood the submission.

Reviewer_mSQNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is a case study on combining the results of multiple models. It focuses on the problem of asymmetric misclassification costs. For example, in a decision problem, the cost of false positives may be much higher than that of false negatives. A simple weighted combination or majority vote of the models’ decisions may then not result in good expected performance. The authors also consider another way to combine the models’ results: they compute a confusion matrix for each model on a calibration dataset and use these confusion matrices to compute the final decision of the system. On three datasets, they show that the confusion-matrix-based method sometimes performs better under the misclassification cost model (Table 1, and Table 5 in the appendix). They also develop PAC-style guarantees to show when a difference in performance can be used to demonstrate that one method dominates the other (Equation 5 and Theorem 4). Finally, they show on which datasets they can certify the gap and on which they cannot.

Critical strengths
  1. The paper identifies the problem of asymmetric costs in model combination.
  2. A well-designed case study illustrates the impact of the problem and suggests an alternative method.
Critical weaknesses
  1. The novelty of the work is limited. Confusion-matrix-based combination has been considered before.

  2. The experiments are limited and are not decisive on all three datasets. I would have expected an analysis on many more datasets.

  3. The cost matrix comes out of the blue. There is no justification for the choice of numbers or for the sets of latent states and actions. The sensitivity to the cost matrix is barely explored. The cost matrix is also tightly coupled to the presentation of the paper.

  4. The cost matrix comes out of the blue. There is no justification for the choice of numbers or for the sets of latent states and actions, and there are no experiments that vary the cost matrix. The cost matrix is also tightly coupled to the presentation of the paper.

  5. The theoretical results are direct consequences of standard theorems.

  6. The paper is poorly written. Here are a few examples: a. As noted above, the cost matrix L is introduced at the beginning of Section 2 without much justification. b. Some symbols in Equation 2 are not defined: π is not clearly defined, and ŷ_k and K are never defined. P_k is explained only implicitly. These are standard notation, but the paper should be self-contained. c. Why is Arm B explained before Arm A? d. While reading Section 4, it was not clear how it relates to Section 2 until reading Table 1, which discusses RWL and uses L. e. What is R in Lemma 2? It is not defined in the main text. f. In Proposition 3, R takes g_c as a parameter. What is g_c, and how does R with a parameter relate to R without one? g. The paper uses colloquial language, such as “ships,” “cosmetic rewrite,” and “fire,” whose meaning is hard to pin down in this context. h. The paper places undue emphasis on the rigor of its experiments. Phrases like “Fidelity is a precondition, not a footnote” add no value to technical writing; ensuring the validity of one’s results is expected anyway.

Questions for authors
  1. Why did you limit your experiments to three datasets?
  2. What is the justification for the chosen action set? How would one obtain realistic values for the loss matrix L in practice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.