- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Paper is about cost-sensitive training of interpolating visual classifiers. Authors clearly differentiate between class imbalance/long-tailed recognition problem versus cost-sensitive training that they study. The proposed method has 2 parts addressing "relative" and "absolute" values in the output, in a sense. Image classification datasets with sparse/dense cost matrices are experimented with to show how the method is performing on minimizing the cost but also some other metrics.
Critical strengths
- The problem is well-identified and the 2-part solution is easy to understand and sensible: first part addresses the relativity in the raw logits in a nonuniform way to care about costs indirectly, and the second part addresses the absoluteness in the softmaxed probabilities to directly care about the overall (expected) cost.
- why it is consequential to your overall assessment: it is a new, interesting way to address the problem.
Critical weaknesses
-
The paper early on claims to be about deep classifiers that interpolate and focus on visual classification (e.g., "TPT" in the title, "visual classification" in the abstract and contributions) but the rest of the paper has little to do with those two things. Yes, there is Definition 1 about TPT but TPT robustness is never used afterwards. The proposed loss modification schemes seem like they are expected to work for a non-interpolating classifier as well as an interpolating one; then why overemphasize TPT? In either case, experiments only mention 1 pretrained classifier at work, which the reader is not even presented proof if it is interpolating the training set (e.g., a near-zero training loss) nor is there any other machine (interpolating again or otherwise) in the experiments to see this distinction paper makes early on. As for the visual part, it seems to be there is nothing visual domain-specific the paper is basing their proposition on, i.e., it seems to be applicable to any domain. In fact, the word "visual" is even dropped at Line 171.
- why it is consequential to your overall assessment: title/abstract/introduction have considerable emphasize on these, but the main text afterwards does not, resulting in confusion for the reader.
-
Experiments are hard to follow and weak: a couple of unfulfilled claims: we read "a CSADA-inspired adversarial adaptation" on Line 308 to see in comparison, but it is not in any of the table of results; "macro-F1" on Line 314 as a metric to report, but it is not found in the results (not even in Appendix). Table 1 and 2 skip the methods from the literature completely, only compares against normal CE and its relatively-low-effort variants. Table 3 has Menon adjustment and Table 4 has Gain-matrix hybrid, but they are not seen together. In terms of the presented performances, Table 3 raises a concern on the 2nd component of the proposition such that its inclusion nearly doubly worsens nATC with significant drops in other metrics; the text explains this by deeming the 2nd component as "an operating-point choice", which does not help in my opinion. Similarly, Table 4 puts NICME behind the vanilla CE significantly (its variants too); this raises a question about the applicability of the proposition.
- why it is consequential to your overall assessment: not orchestrated well to support the proposition empirically, and what is presented already does not really look promising.
Questions for authors
- Why emphasize visual: why can't one use this approach in, say, tabular or text or ... classification? Why emphasize high capacity networks? What prevents one from using this with normal machines, if any?
- Line 52 mentions the loss close to 0, Line 75 mentions misclassification count; why the mismatch?
- Line 70 mentions an aim is to make the loss require a larger raw-logit gap but vanilla cross entropy is already greedy like this, no?
- Line 192: "non-identical and asymmetric": shouldn't this be the other way around?
- Line 212: "subunit" entries weren't in the categorization in its preceding paragraph; what do they correspond to?
- Could the authors expand the justification for "Cost-sensitive regularizer" part of their proposition? e.g., by moving some parts from the referred Appendix E now that there is more space allowed.
- Line 321: "All vision experiments": implies there are other kinds of experiments but are there?
- Could the authors highlight leading results in tables by bold or something upon some statistical test?
Response to Reviewer HSw4
Thank you for recognizing the importance of directed costs. We will foreground margin-only NICME and clarify its applicability and empirical evidence.
Applicability beyond vision
Comment 1: "What prevents one from using this with normal machines, if any?"
Response: The loss applies to any differentiable classifier producing class scores; neither modality nor interpolation is a formal requirement. Vision and ConvNeXt-Base specify the evidence reported here. We will clarify this scope in the title, abstract, and introduction.
Why the directed margin differs from ordinary CE
Comment 2: "Line 70 mentions an aim is to make the loss require a larger raw-logit gap but vanilla cross entropy is already greedy like this, no?"
Response: Yes, CE encourages separation. NICME specifies how much additional separation each directed cost requires. For raw logits z, true class y, and My,k=max(Cy,k,1),
\\ \Longrightarrow\\ z_y-z_k\ge\alpha\log M_{y,k}-\log(e^{\varepsilon}-1).$$ Ordinary CE corresponds to $\alpha=0$. Above the clamp, multiplying one directed cost by q raises its necessary gap by $\alpha\log q$, independently of reverse-direction costs. *Comment 3*: "Line 52 mentions the loss close to 0, Line 75 mentions misclassification count; why the mismatch?" *Response*: We will explicitly distinguish classification error from loss: correct classification only requires the target to have the largest score, while small CE requires stronger confidence. Our new 64-epoch trajectories record both. With augmentation and weight decay retained, all nine CE/margin/full runs sustained clean error ≤0.1% for at least 20 further epochs; eight ended at zero errors. Final clean raw CE ranges approximately from $9\times10^{-5}$ to $4.7\times10^{-4}$. At the shared late checkpoint, margin-only mean raw test nATC was 0.003073 versus CE's 0.004479. These are finite-training diagnostics. **The principal configuration and regularizer** *Comment 4*: "Could the authors expand the justification for "Cost-sensitive regularizer" part of their proposition?" *Response*: We will move the motivation from Appendix E into the main discussion and present margin-only NICME as the primary configuration. The family is $$\ell_{\rm NICME}=\ell_{\rm pcm}+\lambda\sum_k\widetilde C_{y,k}\operatorname{softmax}(z)_k.$$ Here $\widetilde C=C/(C_{\max}+\epsilon_{\rm num})$. The term penalizes unadjusted probability on costly alternatives. Its role is finite-training control; the directed-margin guarantee already holds at $\lambda=0$. We will treat positive regularization as an additional component study. A fresh three-seed balanced PMI-20 comparison shares initial weights, data order, budget, and validation-cost selection. At validation-selected checkpoints, raw nATC is 0.002813 ± 0.000156 for margin-only versus 0.003698 ± 0.001671 for CE. With held-out calibration and cost-min decisions: | Method | nATC | Balanced accuracy | Macro-F1 | |---|---|---|---| | CE | 0.003958 ± 0.001581 | 0.983854 | 0.983883 | | NICME margin-only | 0.002344 ± 0.000681 | 0.985938 | 0.985853 | Costs are mean ± sample SD; other entries are means. These are descriptive operating points across three fresh seeds, distinct from the historical protocols. **Making the literature comparisons and metrics visible** *Comment 5 (excerpts)*: "a CSADA-inspired adversarial adaptation"; "macro-F1" *Response*: CSADA-inspired results are present in Appendix I, Tables 5–7. We will surface them alongside Menon adjustment, the gain hybrid, and calibrated CE, with protocol labels. The new natural-prior panel adds a VS reference control. Numerical macro-F1 and directed counts are now assembled; the table above illustrates their presentation. CSADA-inspired will retain its adaptation label. **Terminology and table presentation** *Comment 6*: "Line 212: "subunit" entries weren't in the categorization in its preceding paragraph; what do they correspond to?" *Response*: They are costs strictly between zero and one. The margin clamps them to the unit baseline; their values still enter the normalized expected-cost term. We will state the cost-unit convention explicitly and replace “non-identical and asymmetric” with “asymmetric pairwise misclassification costs,” defining rows as true labels and columns as predictions. *Comment 7*: "Line 321: "All vision experiments": implies there are other kinds of experiments but are there?" *Response*: We will use “all experiments reported here” rather than the "All vision experiments". There are no other experiments to non-vision domains currently. *Comment 8*: "Could the authors highlight leading results in tables by bold or something upon some statistical test?" *Response*: We will highlight best means with captions identifying them as descriptive, display seed uncertainty, and distinguish that presentation from formal paired-test conclusions.