acceptodds
Under review as a conference paper at ICLR 2027

Beyond Class Imbalance: TPT-Robust Cost-Matrix Embedding for Cost-Sensitive Learning

Abstract

Cost-sensitive classification is critical when prediction errors incur unequal costs, but class-frequency correction and scalar loss reweighting can obscure the directed pairwise structure of the task. A class may be frequent yet safety-critical, and the cost of confusing one class with another need not equal the reverse error. We introduce NICME (Non-Identical Cost-Matrix Embeddings), a cost-sensitive training objective for deep visual classification with non-identical, asymmetric cost matrices. NICME embeds each true-label row of the cost matrix into logit space through pairwise cost margins, together with a normalized expected-cost regularizer on the original predictive distribution. An exact finite-loss characterization shows how above-baseline costs increase the raw-logit separation required from costly wrong classes, including at arbitrarily small positive loss levels. The objective retains ordinary raw-logit inference. Experiments combine balanced binary and multiclass tasks, calibrated minimum-cost decision baselines, an eight-seed component study, and six dense asymmetric cost matrices on CIFAR-100. NICME provides favorable cost–utility operating points on the retained PMI-20 and BreaKHis comparisons and lowers mean raw-argmax cost in five of the six CIFAR-100 settings. Component and sensitivity studies identify the directed margin as a useful mechanism and characterize the role of regularization, calibration, and cost scale.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
cost-sensitive learning, deep learning, computer vision, asymmetric cost matrices

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HSw4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Paper is about cost-sensitive training of interpolating visual classifiers. Authors clearly differentiate between class imbalance/long-tailed recognition problem versus cost-sensitive training that they study. The proposed method has 2 parts addressing "relative" and "absolute" values in the output, in a sense. Image classification datasets with sparse/dense cost matrices are experimented with to show how the method is performing on minimizing the cost but also some other metrics.

Critical strengths
  1. The problem is well-identified and the 2-part solution is easy to understand and sensible: first part addresses the relativity in the raw logits in a nonuniform way to care about costs indirectly, and the second part addresses the absoluteness in the softmaxed probabilities to directly care about the overall (expected) cost.
    • why it is consequential to your overall assessment: it is a new, interesting way to address the problem.
Critical weaknesses
  1. The paper early on claims to be about deep classifiers that interpolate and focus on visual classification (e.g., "TPT" in the title, "visual classification" in the abstract and contributions) but the rest of the paper has little to do with those two things. Yes, there is Definition 1 about TPT but TPT robustness is never used afterwards. The proposed loss modification schemes seem like they are expected to work for a non-interpolating classifier as well as an interpolating one; then why overemphasize TPT? In either case, experiments only mention 1 pretrained classifier at work, which the reader is not even presented proof if it is interpolating the training set (e.g., a near-zero training loss) nor is there any other machine (interpolating again or otherwise) in the experiments to see this distinction paper makes early on. As for the visual part, it seems to be there is nothing visual domain-specific the paper is basing their proposition on, i.e., it seems to be applicable to any domain. In fact, the word "visual" is even dropped at Line 171.

    • why it is consequential to your overall assessment: title/abstract/introduction have considerable emphasize on these, but the main text afterwards does not, resulting in confusion for the reader.
  2. Experiments are hard to follow and weak: a couple of unfulfilled claims: we read "a CSADA-inspired adversarial adaptation" on Line 308 to see in comparison, but it is not in any of the table of results; "macro-F1" on Line 314 as a metric to report, but it is not found in the results (not even in Appendix). Table 1 and 2 skip the methods from the literature completely, only compares against normal CE and its relatively-low-effort variants. Table 3 has Menon adjustment and Table 4 has Gain-matrix hybrid, but they are not seen together. In terms of the presented performances, Table 3 raises a concern on the 2nd component of the proposition such that its inclusion nearly doubly worsens nATC with significant drops in other metrics; the text explains this by deeming the 2nd component as "an operating-point choice", which does not help in my opinion. Similarly, Table 4 puts NICME behind the vanilla CE significantly (its variants too); this raises a question about the applicability of the proposition.

    • why it is consequential to your overall assessment: not orchestrated well to support the proposition empirically, and what is presented already does not really look promising.
Questions for authors
  • Why emphasize visual: why can't one use this approach in, say, tabular or text or ... classification? Why emphasize high capacity networks? What prevents one from using this with normal machines, if any?
  • Line 52 mentions the loss close to 0, Line 75 mentions misclassification count; why the mismatch?
  • Line 70 mentions an aim is to make the loss require a larger raw-logit gap but vanilla cross entropy is already greedy like this, no?
  • Line 192: "non-identical and asymmetric": shouldn't this be the other way around?
  • Line 212: "subunit" entries weren't in the categorization in its preceding paragraph; what do they correspond to?
  • Could the authors expand the justification for "Cost-sensitive regularizer" part of their proposition? e.g., by moving some parts from the referred Appendix E now that there is more space allowed.
  • Line 321: "All vision experiments": implies there are other kinds of experiments but are there?
  • Could the authors highlight leading results in tables by bold or something upon some statistical test?
1 comment
AuthorsOpenReview ↗

Response to Reviewer HSw4

Thank you for recognizing the importance of directed costs. We will foreground margin-only NICME and clarify its applicability and empirical evidence.

Applicability beyond vision

Comment 1: "What prevents one from using this with normal machines, if any?"

Response: The loss applies to any differentiable classifier producing class scores; neither modality nor interpolation is a formal requirement. Vision and ConvNeXt-Base specify the evidence reported here. We will clarify this scope in the title, abstract, and introduction.

Why the directed margin differs from ordinary CE

Comment 2: "Line 70 mentions an aim is to make the loss require a larger raw-logit gap but vanilla cross entropy is already greedy like this, no?"

Response: Yes, CE encourages separation. NICME specifies how much additional separation each directed cost requires. For raw logits z, true class y, and My,k=max⁡(Cy,k,1)M_{y,k}=\max(C_{y,k},1),

\\ \Longrightarrow\\ z_y-z_k\ge\alpha\log M_{y,k}-\log(e^{\varepsilon}-1).$$ Ordinary CE corresponds to $\alpha=0$. Above the clamp, multiplying one directed cost by q raises its necessary gap by $\alpha\log q$, independently of reverse-direction costs. *Comment 3*: "Line 52 mentions the loss close to 0, Line 75 mentions misclassification count; why the mismatch?" *Response*: We will explicitly distinguish classification error from loss: correct classification only requires the target to have the largest score, while small CE requires stronger confidence. Our new 64-epoch trajectories record both. With augmentation and weight decay retained, all nine CE/margin/full runs sustained clean error ≤0.1% for at least 20 further epochs; eight ended at zero errors. Final clean raw CE ranges approximately from $9\times10^{-5}$ to $4.7\times10^{-4}$. At the shared late checkpoint, margin-only mean raw test nATC was 0.003073 versus CE's 0.004479. These are finite-training diagnostics. **The principal configuration and regularizer** *Comment 4*: "Could the authors expand the justification for "Cost-sensitive regularizer" part of their proposition?" *Response*: We will move the motivation from Appendix E into the main discussion and present margin-only NICME as the primary configuration. The family is $$\ell_{\rm NICME}=\ell_{\rm pcm}+\lambda\sum_k\widetilde C_{y,k}\operatorname{softmax}(z)_k.$$ Here $\widetilde C=C/(C_{\max}+\epsilon_{\rm num})$. The term penalizes unadjusted probability on costly alternatives. Its role is finite-training control; the directed-margin guarantee already holds at $\lambda=0$. We will treat positive regularization as an additional component study. A fresh three-seed balanced PMI-20 comparison shares initial weights, data order, budget, and validation-cost selection. At validation-selected checkpoints, raw nATC is 0.002813 ± 0.000156 for margin-only versus 0.003698 ± 0.001671 for CE. With held-out calibration and cost-min decisions: | Method | nATC | Balanced accuracy | Macro-F1 | |---|---|---|---| | CE | 0.003958 ± 0.001581 | 0.983854 | 0.983883 | | NICME margin-only | 0.002344 ± 0.000681 | 0.985938 | 0.985853 | Costs are mean ± sample SD; other entries are means. These are descriptive operating points across three fresh seeds, distinct from the historical protocols. **Making the literature comparisons and metrics visible** *Comment 5 (excerpts)*: "a CSADA-inspired adversarial adaptation"; "macro-F1" *Response*: CSADA-inspired results are present in Appendix I, Tables 5–7. We will surface them alongside Menon adjustment, the gain hybrid, and calibrated CE, with protocol labels. The new natural-prior panel adds a VS reference control. Numerical macro-F1 and directed counts are now assembled; the table above illustrates their presentation. CSADA-inspired will retain its adaptation label. **Terminology and table presentation** *Comment 6*: "Line 212: "subunit" entries weren't in the categorization in its preceding paragraph; what do they correspond to?" *Response*: They are costs strictly between zero and one. The margin clamps them to the unit baseline; their values still enter the normalized expected-cost term. We will state the cost-unit convention explicitly and replace “non-identical and asymmetric” with “asymmetric pairwise misclassification costs,” defining rows as true labels and columns as predictions. *Comment 7*: "Line 321: "All vision experiments": implies there are other kinds of experiments but are there?" *Response*: We will use “all experiments reported here” rather than the "All vision experiments". There are no other experiments to non-vision domains currently. *Comment 8*: "Could the authors highlight leading results in tables by bold or something upon some statistical test?" *Response*: We will highlight best means with captions identifying them as descriptive, display seed uncertainty, and distinguish that presentation from formal paired-test conclusions.
Reviewer_ptPeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that pairwise, asymmetric cost sensitivity is distinct from class imbalance, since frequency-based corrections become constant under a balanced prior while a directed cost matrix does not. It also argues that scalar cost reweighting loses its effect once overparameterized networks interpolate the training data, the terminal phase of training (TPT). The proposed method, NICME, adds a cost-dependent offset αlog⁡max⁡(Cy,k,1)\alpha\log\max(C_{y,k},1) to each wrong-class logit before cross-entropy, plus a normalized expected-cost regularizer on the unadjusted softmax. Proposition 1 shows that a loss at most ε\varepsilon requires zy−zk≥αlog⁡My,k−log⁡(eε−1)z_y - z_k \ge \alpha\log M_{y,k} - \log(e^\varepsilon - 1). Experiments fine-tune ConvNeXt-Base on balanced PMI-20, two binary tasks (Spider, BreaKHis) and six synthetic dense CIFAR-100 cost matrices, with calibration baselines, an eight-seed ablation, a matched search and sensitivity studies. The authors report lower normalized cost than calibrated CE on PMI-20 and BreaKHis, and in 5 of 6 CIFAR-100 settings.

Critical strengths

S1. Clear separation of cost sensitivity from class imbalance. The paper shows that frequency-derived weights and prior-based logit offsets reduce exactly to CE under a balanced prior, while directed costs do not. Menon adjustment and CE are identical on BreaKHis (Table 7). Balanced-class evaluation as a diagnostic for whether a method actually uses the cost matrix is a useful, transferable contribution.

S2. Transparent reporting. The paper includes the strongest baseline, calibrated CE with Bayes cost-min decisions. It also retains all seeds, predeclares paired tests, reports null results and provides code with saved logits. This makes the claims auditable, and easier to reproduce.

Critical weaknesses

W1. "TPT robustness" does not address the motivating failure mode, and the regularizer collapses in TPT.

  • The cost term fades. In Proposition 1 the cost term αlog⁡My,k\alpha\log M_{y,k} is a fixed constant, while −log⁡(eε−1)→∞-\log(e^\varepsilon-1)\to\infty as training loss goes to zero. In the interpolating regime the cost contribution therefore becomes negligible relative to the overall margin, and the asymptotic decision boundary need not reflect CC. This matches Kini et al. (2021), cited by the paper, who show that additive logit adjustments lose their effect in the separable regime. Definition 1 concedes this point.
  • The regularizer stops working. Its value and gradient, pq(C~y,q−rC)p_q(\tilde C_{y,q} - r_C), both go to zero as the softmax becomes one-hot. This is the same collapse the paper attributes to the expected-cost surrogate in Eq. 4.
  • TPT is never shown to occur. No training accuracy or training-loss curves are reported. All models are fine-tuned from pretrained weights with early stopping (patience 3–5), so the TPT motivation may not apply to the experiments at all.

W2. The controlled evidence does not support the proposed objective.

  • Eight-seed ablation (Tables 3, 14). Full NICME is indistinguishable from CE (paired difference +0.0002+0.0002, p=0.98p = 0.98) and has lower mean balanced accuracy. The best variant is margin-only (λ=0\lambda = 0), so the regularizer appears to hurt.
  • Matched search (Table 4). NICME (0.01036) is about twice as costly as CE (0.00510) and calibrated CE (0.00490).
  • Natural-prior PMI-20 (Table 23). This is the only experiment with both class imbalance and directed costs. NICME (0.00370) and NICME + prior (0.00431) both lose to CE (0.00232) and calibrated CE (0.00136). This would be the common real-world setting where both class imbalance, and cost-sensitive learning where the proposed approach NICME performs worse than pure CE. Understanding why, or mitigating this would be extremely useful, and make a strong point about the efficacy of the approach.
  • CIFAR-100 (Tables 17–18). Calibrated CE with cost-min decisions beats NICME's raw argmax in 5 of 6 settings. The main text compares argmax against argmax only.
  • Headline results. These come from three "retained" runs whose seeds may not be independent (App. H.2). The retained NICME result on PMI-20 (0.00370) is about half the controlled result for the same configuration (0.00723). No significance tests are given for Tables 1 and 2. On PMI-20 the Table 1 gap amounts to roughly one critical error per run.

Additional Concerns

  • W3. Novelty. The margin term is Menon et al.'s logit adjustment with pairwise offsets, close to the gain-matrix logit adjustment of Narasimhan & Menon (2021). Proposition 1 follows from bounding one summand. There is no comparison with multiplicative/VS-style pairwise adjustments or Bayes-consistent cost-sensitive surrogates.
  • W4. Baselines. Several look untuned or collapsed: cost-weighted CE has BA 0.509 on Spider, the CSADA-inspired method has BA 0.504 on BreaKHis, and the gain hybrid has BA 0.59 ± 0.46. On CIFAR-100, NICME checkpoints are selected on validation nATC while the CE anchors appear not to be.
  • W5. Target-min recall. It does not distinguish costly from cheap confusions. It is also the checkpoint-selection criterion on PMI-20, it moves in steps of about 0.03 (around 32 test images per class), and it saturates on validation (45 of 49 grid points reach 1.0). Critical-pair error counts would be more informative.
  • W6. Figure 1. Raw margins depend on overall logit scale, so a lower effective temperature shifts the CDF right without changing decisions. NICME is also trained to enlarge exactly these margins. Normalized or temperature-scaled margins, or margins relative to non-critical pairs, would be needed to show targeted separation. Some more information about what is the "empirical cumulative probability (y-axis)" in Figure 1 would be useful, as it current stands unexplained.
Questions for authors
  • Since αlog⁡My,k\alpha\log M_{y,k} is constant while the required margin diverges, in what sense is NICME TPT-robust beyond what any additive adjustment satisfies? How does this relate to Kini et al. (2021)?
  • Do the selected checkpoints actually reach the interpolating regime? Please report training accuracy and loss.
  • Since the regularizer vanishes in TPT and hurts in Table 3, why is the regularizer included in the full objective of the proposed method? The logit shift also forces the model to learn better decision boundaries, and helps reduce the costly mistakes.
  • Which PMI-20 result represents NICME's performance: retained (0.00370), controlled (0.00723) or matched-search (0.01036)?
  • On CIFAR-100, why not compare against calibrated CE cost-min decisions, and were the CE checkpoints selected with the same nATC criterion?
  • Can you add paired significance tests for Tables 1–2?
  • Were cost-weighted CE, the CSADA-inspired method and the gain hybrid tuned comparably to NICME?
  • Can Figure 1 be shown with temperature-scaled or normalized margins?

Suggestions: Reframe the contribution around the margin-only variant under a single predeclared protocol, and move a genuinely imbalanced, cost-sensitive benchmark into the main text.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ptPe

Thank you for recognizing the balanced diagnostic. We adopt your suggestion to foreground margin-only NICME and the natural-prior benchmark.

Finite-loss geometry and terminal-phase behavior

Comment 1: "How does this relate to Kini et al. (2021)?"

Response: Kini et al.'s asymptotic separator analysis and NICME's finite-loss statement address different quantities. With My,k=max⁡(Cy,k,1)M_{y,k}=\max(C_{y,k},1),

\\ \Longrightarrow\\ z_y-z_k\ge\alpha\log M_{y,k}-\log(e^{\varepsilon}-1).$$ The additional directed gap is explicit at every positive tolerance. We will describe this as a finite-loss margin property, with the regularizer as an optional finite-training component. We now have 64-epoch CE/margin/full trajectories retaining augmentation and weight decay. All nine sustained clean error ≤0.1% for at least 20 further epochs; eight finished with zero errors. At the common epoch-64 checkpoint, margin-only mean raw test nATC was 0.003073 versus CE's 0.004479, with larger normalized critical margins in every seed. Clean training CE, classification errors, and regularizer-gradient trajectories are recorded separately. This is finite-training evidence, not an asymptotic separator guarantee. **A clear primary NICME configuration** *Comment 2 (excerpt)*: "Which PMI-20 result represents NICME's performance" *Response*: The three results belong to different protocols. We will foreground $\lambda=0$ NICME and the fresh controlled comparison below. Across fresh seeds 60–62, with identical initial weights/data order and validation-cost selection, validation-selected balanced PMI-20 results are: | Decision | CE nATC | NICME margin-only nATC | |---|---|---| | Raw argmax | 0.003698 ± 0.001671 | 0.002813 ± 0.000156 | | Calibrated cost-min | 0.003958 ± 0.001581 | 0.002344 ± 0.000681 | Values are mean ± sample SD. The mean cost reductions are approximately 24% and 41%. Calibrated costly-pair errors decrease from 1.67 to 0.67 per run. Cost and directed counts are primary endpoints; target recall is a utility measure. Positive regularization will be an additional study, not a prerequisite for NICME's margin mechanism. **Natural priors and additional controls** *Comment 3 (excerpt)*: "move a genuinely imbalanced, cost-sensitive benchmark into the main text." *Response*: We completed a matched natural-prior PMI-20 panel. With held-out calibration and cost-min decisions, margin-only nATC is 0.000586 ± 0.000279 versus CE's 0.000682 ± 0.000375; target-min recall is 98.63% versus 96.75%. This identifies a useful calibrated operating point when both effects are present. The same panel includes prior adjustment and a VS reference setting; their mean calibrated costs are 0.000754 and 0.001436. For novelty, we will explicitly relate the margin to Gimpel–Smith softmax-margin with directed log-cost augmentation and distinguish prior-only adjustment from general pairwise/gain-matrix surrogates. The contribution is the concrete construction, finite-loss interpretation, and diagnostic evaluation. **Calibrated baselines and checkpoint parity** *Comment 4*: "On CIFAR-100, why not compare against calibrated CE cost-min decisions, and were the CE checkpoints selected with the same nATC criterion?" *Response*: Both-calibrated comparisons are already included in the six-matrix analysis; we will make them more prominent. The original CE anchors were cost-selected for up-r2 and reused across matrices. We completed a fresh eight-epoch, three-seed component panel on one prechosen matrix, giving CE and NICME identical validation-cost selection with BA ≥0.70. Calibration uses a disjoint partition after checkpoint identities are frozen. Every new method is evaluated under both decision rules. **Figure 1 and uncertainty** *Comment 5*: "Can Figure 1 be shown with temperature-scaled or normalized margins?" *Response*: Yes. We replot the retained CE/full-NICME models using $(z_y-z_k)/\operatorname{sd}(z)$ and temperature-scaled margins, with ordinary pairs on the same source images. Normalized critical means are 2.61 for CE and 3.61 for NICME; ordinary means are 3.76 and 3.98. Global positive scaling alone cannot explain this targeted descriptive shift. The ECDF y-axis is the fraction of image/pair observations with margin ≤ the horizontal-axis value. *Comment 6*: "Can you add paired significance tests for Tables 1–2?" *Response*: We computed conditional paired-bootstrap intervals for the retained models, clustering BreaKHis by patient. For retained PMI-20 raw full-NICME-minus-CE nATC, the conditional 95% interval is [-0.003960, -0.000313]. This measures sample uncertainty conditional on fitted models, not training-seed significance. New seed comparisons remain descriptive. We will display historical tuning settings, including the gain hybrid's nine-candidate search, alongside the new panels' shared budgets and selection criteria.
Reviewer_6pEPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes NICME for multiclass classification with asymmetric, pairwise misclassification costs. The objective combines a cost-dependent margin loss with a normalized expected-cost regularizer. During training, the margin component adds offsets to non-target logits according to the corresponding entries of the cost matrix. Achieving a small loss consequently requires larger raw-logit separation from high-cost incorrect classes. The regularizer additionally penalizes probability mass assigned to costly predictions. Inference uses the original logits and does not require label-dependent adjustments.

The theoretical contribution characterizes the necessary score separation at any positive loss tolerance. The empirical evaluation covers balanced PMI-20, Spider, BreaKHis, and CIFAR-100 under six dense asymmetric cost matrices. Additional analyses examine calibrated minimum-cost decisions, component ablations, hyperparameter selection, cost-scale sensitivity, and composition with class-prior correction.

Critical strengths
  1. A practical formulation for directed error costs. The method preserves the pairwise structure of the cost matrix, including differences between opposite error directions and between errors originating from the same class. The formulation is compatible with standard cross-entropy training infrastructure and ordinary raw-logit inference, with no adversarial inner optimization. The balanced-data protocol is useful for isolating this cost-information channel from class-frequency correction. These features make the approach straightforward to implement and evaluate.

  2. A substantial evaluation with explicit treatment of uncertainty and negative results. The paper includes calibrated decision baselines, an eight-seed component study, dense multiclass cost matrices, and sensitivity analyses. It retains adverse runs and explicitly reports that the controlled tests do not establish superiority of the full objective. The CIFAR-100 study provides positive evidence: NICME achieves lower mean normalized cost in five of six raw-argmax comparisons and four of six calibrated minimum-cost comparisons. Reporting balanced accuracy and directed error counts alongside cost makes the associated tradeoffs assessable. These are meaningful strengths, even though the comparisons remain descriptive results on fixed splits.

Critical weaknesses
  1. The methodological advance over existing cost-augmented losses is insufficiently established.

    The margin component has the form of a cost-augmented softmax loss, with the augmentation specified by a scaled logarithm of the clamped cost entry. At the loss-function level, this is a specialization of the softmax-margin construction described by Gimpel and Smith (2010). This relationship is not discussed in the manuscript. The paper does discuss Narasimhan and Menon (2021), which already develops cost-sensitive logit adjustments for overparameterized models. The distinct contribution therefore needs to be established more precisely through the logarithmic cost transformation, its combination with expected-cost regularization, or a demonstrated advantage in the proposed setting.

    Proposition 1 gives a valid and interpretable finite-loss bound, but follows directly from the loss construction. Definition 1 explicitly limits the result to this property, so the absence of an asymptotic or Bayes-consistency guarantee is not a contradiction. Nevertheless, the result does not establish a persistent cost-sensitive decision boundary or a generalization advantage during the terminal phase of training. Figure 1 reports final test-margin distributions, and the experiments use early stopping; neither directly characterizes behavior after training error reaches zero. Consequently, the connection between the finite-loss property and the practical terminal-phase motivation remains insufficiently demonstrated.

  2. The controlled results do not yet justify the reference combination or establish when it should be preferred to simpler alternatives.

    Table 3 is particularly important because its eight-seed comparison uses a shared training protocol and selection criterion. Mean normalized cost is 0.00703 for cross-entropy, 0.00410 for the margin-only variant, and 0.00723 for full NICME. The full objective also has greater variability and lower mean balanced accuracy than cross-entropy. Appendix K reports a paired confidence interval spanning improvements and degradations; these results do not establish either superiority or equivalence. The calibrated analysis in Table 16 similarly favors the margin-only variant in mean cost, while full NICME has higher mean cost than calibrated cross-entropy.

    The separate search in Table 4 provides no stronger support for the reference combination: its mean normalized cost is 0.01036, compared with 0.00510 for cross-entropy and 0.00490 for calibrated cross-entropy with minimum-cost decisions. Appendix H.2 acknowledges that matching the number of candidate configurations does not match the searched dimensions or checkpoint-selection criteria. The favorable historical comparisons should also receive less evidential weight because their runs were not verified as independent training seeds.

    The margin-only variant remains a legitimate and promising member of the NICME family. The unresolved issue is the incremental value of the regularizer and the reliability of the proposed selection procedure. A component comparison on the dense CIFAR-100 settings would help attribute the positive results there. It is also important to clarify whether the cross-entropy anchor checkpoints received the same cost-aware selection opportunity as NICME, which is selected separately for each cost matrix. These questions materially affect the strength of the practical contribution.

Questions for authors
  1. Methodological distinction. How should NICME’s margin component be distinguished from softmax-margin training with a logarithmically transformed cost function? Is the principal contribution the transformation, the combination with regularization, or its empirical behavior in deep visual classification? A direct comparison under matched margin scales would help establish the significance of that distinction.

  2. Contribution of the regularizer. Which controlled results establish a benefit of the full objective over margin-only training? In particular, are margin-only results available for the six CIFAR-100 cost matrices under identical training and checkpoint-selection criteria? If the benefit is specific to certain settings, what validation evidence supports identifying those settings in advance?

  3. Checkpoint selection and baseline comparability. What criterion selected the CIFAR-100 cross-entropy anchors? Would the conclusions change if checkpoints from the same cross-entropy training trajectories were selected separately for each matrix using NICME’s validation-cost criterion and accuracy constraint? Please also specify the checkpoint-selection differences in the matched-search experiment.

  4. Connection to terminal-phase training. Do the evaluated models reach zero training classification error before the selected checkpoint? Are training trajectories available showing whether directed-margin and held-out-cost improvements persist after interpolation? This would help distinguish the stated finite-loss property from a demonstrated terminal-phase advantage.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6pEP

Response to Reviewer 6pEP

Thank you for recognizing the practicality of directed costs, the balanced diagnostic, and the promise of margin-only NICME. We will foreground that configuration and make the contribution more explicit.

Methodological distinction

Comment 1: "How should NICME’s margin component be distinguished from softmax-margin training with a logarithmically transformed cost function?"

Response: NICME instantiates softmax-margin with a concrete directed log-cost transformation. For raw logits z, true label y, and My,k=max⁡(Cy,k,1)M_{y,k}=\max(C_{y,k},1),

ℓpcm=log⁡(1+∑k≠yMy,kαezk−zy).\ell_{\rm pcm}=\log\left(1+\sum_{k\ne y}M_{y,k}^{\alpha}e^{z_k-z_y}\right).

Thus, costs enter as power weights inside the exponential sum. Our contribution is this explicit cost-to-margin design, its entry-wise finite-loss interpretation, and evaluation separating directed costs from class frequency. We will cite Gimpel and Smith (2010) and position NICME within that framework. Menon's usual prior adjustment cancels under balanced priors; broader pairwise-offset and Narasimhan–Menon gain-matrix losses also encode asymmetric costs. NICME provides a simple construction compatible with standard CE training and raw-logit inference.

Margin-only NICME and empirical support

Comment 2: "Which controlled results establish a benefit of the full objective over margin-only training?"

Response: We will center the practical contribution on NICME with λ=0\lambda=0, already a member of the proposed family, and present positive regularization as an additional component study. The directed-margin mechanism and Proposition 1 do not require the regularizer.

We completed a fresh three-seed balanced PMI-20 study with identical initial weights, data order, training budget, and validation-cost selection. At validation-selected checkpoints:

DecisionCE nATCNICME margin-only nATC
Raw argmax0.003698 ± 0.0016710.002813 ± 0.000156
Calibrated cost-min0.003958 ± 0.0015810.002344 ± 0.000681

Values are mean ± sample SD; lower is better. These are approximately 24% and 41% lower mean costs, respectively. The original eight-seed study additionally favors margin-only over CE in 7/8 raw-cost comparisons. These are descriptive results on fixed splits.

We also completed a three-seed CE/margin/full component panel on one prechosen dense CIFAR matrix, with eight epochs per method, plus a mean-offset-matched linear-cost comparator. These extend the component analysis; they are distinct from the original six-matrix study.

Checkpoint selection and comparability

Comment 3: "What criterion selected the CIFAR-100 cross-entropy anchors?"

Response: The original anchors were selected by validation nATC under BA ≥0.70 for the original up-r2 matrix, then reused across matrices. The fresh CIFAR panel gives every method identical cost-aware selection on its study matrix. PMI-20 selection likewise minimizes validation cost subject to target-min recall ≥0.90 and BA ≥0.80. These are validation constraints. All checkpoint identities were frozen before calibration/test evaluation.

Our audit also clarifies Appendix H.2: all four selected matched-search families used target-min recall for within-run checkpoint selection, followed by common cost-aware outer configuration ranking. Their search dimensions differed. We will state this distinction precisely.

Behavior after interpolation

Comment 4: "Are training trajectories available showing whether directed-margin and held-out-cost improvements persist after interpolation?"

Response: Yes. We ran fresh CE, margin-only, and full-NICME trajectories for 64 epochs, retaining augmentation and weight decay. All nine sustained clean training error ≤0.1% for at least 20 further epochs; eight finished with zero errors.

At the common epoch-64 endpoint, after every method had entered its sustained near-zero regime, margin-only mean raw test nATC was 0.003073 versus CE's 0.004479. Its normalized critical test margins exceeded CE's in every seed. These trajectory diagnostics are separate from the validation-selected results above.

The finite-loss mechanism is explicit:

\\ \Longrightarrow\\ z_y-z_k\ge\alpha\log M_{y,k}-\log(e^{\varepsilon}-1).$$ Above the clamp, multiplying a cost by q adds $\alpha\log q$ to the required gap at every positive loss tolerance. We will connect this finite-training interpretation to the new trajectories and distinguish it from asymptotic boundary or Bayes-consistency guarantees.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.