acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Harmonic Loss via Non-Euclidean Distance Layers

Abstract

Cross-entropy is the standard objective for training deep networks, yet it gives no inherent meaning to the learned weight vectors and drives them to grow without bound. The harmonic loss is a distance-based alternative that improves interpretability and mitigates grokking, but it has been studied only with the Euclidean distance, and never for its computational or energy cost. We replace that distance with a broad spectrum of metrics and evaluate the resulting distance-tailored harmonic losses on vision backbones and language models along three axes: model performance, interpretability, and sustainability. Across vision backbones, replacing the Euclidean distance is close to free: Bray-Curtis, cosine and Mahalanobis heads improve on Euclidean harmonic loss in most of the dataset-backbone settings we measure, and on the deeper backbones also on cross-entropy, at comparable measured emissions; where the harmonic family trails cross-entropy, on shallow backbones, the choice of distance still recovers most of the gap. On language modeling tasks, for smaller models, cosine-based harmonic losses improve gradient and learning stability, strengthen representation structure, and reduce emissions relative to cross-entropy and Euclidean harmonic loss. For a larger, modern decoder (SmolLM2), Euclidean harmonic loss attains the lowest perplexity and a Hamming-based harmonic loss the most structured representations at matched perplexity, at a higher training cost. On a pretrained 7B model, whose head spans only a few hundred classes, that cost disappears: harmonic heads match cross-entropy's accuracy while improving representation structure and out-of-distribution detection at no additional training cost. Our code is available at: https://anonymous.4open.science/r/rethinking-harmonic-loss/.

Primary area:
interpretability and explainable AI
Keywords:
harmonic loss, distance metrics, neural networks, loss functions, deep learning, interpretability, green AI, computational efficiency, cross-entropy, cosine distance

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1JGdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on how distance choice affects harmonic loss by extending the original Euclidean formulation to alternatives such as cosine, Manhattan, Mahalanobis, and differentiable Hamming variants. It compares predictive performance, representation structure, and training cost across vision and language tasks, including SmolLM2 pretraining and classification with a pretrained 7B model. It also examines when these alternatives improve accuracy, produce more structured representations, or offer favorable computational trade-offs.

Critical strengths

Broad empirical evaluation. The paper compares multiple distances across vision and language tasks, with additional loss baselines, repeated seeds, and frozen-backbone and exponent ablations.

Careful analysis of confounders and limitations. Matched-perplexity comparisons help separate representation concentration from predictive quality. The paper also clarifies theoretical limits and reports failed configurations and computational overhead.

Useful insights into scaling. The study shows that head rankings and computational costs change with task and output-space size, clarifying why gains in small models or classification may not transfer to vocabulary-scale language modeling.

Critical weaknesses

The explainability evidence is limited. The PCA concentration and silhouette coefficient can reflect the representation structure, but they cannot directly prove that the features or predictions are easier to be understood by people.

There are confounding factors in comparison. The differences in effective index, cosine form and numerical processing make it difficult to distinguish the independent effects of distance selection. Out-of-distribution detection also lacks a calibrated cross-entropy baseline.

The method innovation is limited. The main extension is to replace the mature distance function in the existing harmonic loss framework. The experimental research is valuable, but the guidance on when and why to choose a certain distance is still relatively limited.

The actual benefits are not stable. The visual comparison has not been corrected by multiple test corrections. The Euclidean distance achieves the best SmolLM2 perplexity, while the representation improvement of soft Hamming is accompanied by significant computational overhead.

Questions for authors

i. Under matched effective exponents and stable log-space implementations, which gains remain attributable to non-Euclidean geometry rather than exponent choice, cosine probability mappings, or numerical safeguards?

ii. Beyond matched-quality PCA and silhouette scores, what demonstrates improved interpretability? Given the weaker prototype–class-mean alignment, can the authors show a concrete task where these heads provide more faithful or useful explanations?

iii. Given the non-significant corrected vision comparisons, Euclidean’s best SmolLM2 perplexity, and Hamming’s overhead, what reproducible advantage remains over well-tuned Euclidean harmonic loss? Can a distance-selection rule independent of test results generalize to held-out settings under comparable tuning and compute budgets?

iv. Do the OOD gains persist against validation-calibrated cross-entropy and tuned cosine-softmax? What separates the contribution of non-Euclidean geometry from confidence calibration or the general benefit of harmonic heads?

2 comments

Thank you for recognizing the breadth of the experiments and the value of comparing representations at matched perplexity. We also appreciate your questions about which benefits remain after tuning and what the representation measures mean in practice. We have revised the claims and added the following checks:

i) Distance, exponent, and implementation. We used the same stable log-space formula and the same validation search over learning rates and exponents for all harmonic heads. After tuning, cosine harmonic needed 5.0 principal components on average to explain 90% of the variance, compared with 9.3 for Euclidean harmonic, while achieving similar test accuracy (93.18% vs. 92.83%). Normalized Bray–Curtis also needed 5.0 components and scored 0.63 percentage points below Euclidean. This gives a practical reason to include cosine in the validation search when representation concentration matters and accuracy needs to stay close to the Euclidean reference. The result applies to the validation-tuned configurations under a common numerical implementation. We hope that this addresses your concern.

ii) Prototype examples. Thank you for the interesting suggestion. Based on your comment, we added real CLINC150 training utterances retrieved for six intents chosen before inspecting the results. For bill balance, cosine harmonic ranked “how much is the total due for my pgw gas bill” first. Euclidean ranked “what questions do you respond to” first, which belongs to another intent. Across all six intents, cosine harmonic, soft Hamming harmonic, cross-entropy, and cosine-softmax each retrieved 30 of 30 examples with the matching label. Euclidean matched 16 of 30. The appendix shows all first-ranked examples in detail. We use “representation structure” for PCA and silhouette and report the examples as a direct check of what the heads retrieve.

iii) Head transferability across datasets. On SVHN, we gave all heads the same 50-epoch budget, tuned on validation data, and froze the choice before testing on three fresh seeds. The rule chose cosine harmonic (learning rate 0.3, exponent 1). Mean test accuracies were 96.99 ± 0.16% for cosine, 96.68 ± 0.06% for Euclidean, and 96.26 ± 0.13% for cross-entropy. The selected head gained +0.31 ± 0.21 points over tuned Euclidean (paired mean ± SD, 95% interval −0.21 to +0.84). The 200-epoch CIFAR-10 and CIFAR-100 controls likewise put cosine ahead of tuned Euclidean by +0.32 ± 0.14 and +0.57 ± 0.03 points, respectively. Although the mean accuracy gains can be regarded as modest, on CIFAR-10, cosine harmonic also concentrated the representation into fewer principal components while achieving similar accuracy to tuned Euclidean. The CLINC150 examples give readers a separate way to inspect what the learned heads retrieve. When considered in combination, these results give a practical reason to include cosine in the validation search. Our guidance keeps Euclidean as a reference, uses validation accuracy to identify competitive heads, and then considers representation structure and compute cost. We include this in our paper as a practical takeaway for researchers and practitioners confirmed by several experiments.

iv) Calibrated and tuned baselines. We selected the cosine-softmax learning rates on CLINC150 validation data and fitted a temperature for every head as suggested. Across five seeds, calibrated AUROC and false-positive rate at 95% detection were 0.971 / 13.3% for cross-entropy and 0.983 / 6.7% for tuned cosine-softmax. Euclidean, cosine, and soft Hamming harmonic reached 0.983 / 6.1%, 0.984 / 5.8%, and 0.985 / 5.8%. Something noteworthy here is that the gains over cross-entropy persist after calibration. The revised Language section and appendix give the full comparison.

We are continuing the follow-up experiments and would welcome additional thoughts if any. We hope that the new experiments and our revised claims address your main concerns, and we would be grateful if you would consider raising your score in support of our paper.

Thank you for asking how to choose a head on a new dataset. We tested our validation-based selection rule on SVHN. All three heads received the same 50-epoch training budget. We tuned each head on validation data and fixed the choice before evaluating on three fresh matched seeds. Mean test accuracy was 96.99% for cosine, 96.68% for tuned Euclidean, and 96.26% for cross-entropy. The paired gain over tuned Euclidean was 0.31 percentage points (95% interval −0.21 to +0.84). This gives a concrete example of how to use the rule, with uncertainty about the size of the gain. Table 48 in the revised paper gives the selected settings and full results. We hope this addresses your main concerns, and we would be grateful if you would reconsider your score.

Reviewer_Sz5xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores alternatives to cross-entropy for classification. Specifically, harmonic losses are explored, with a variety of distance functions between the instance representation and the class prototype. The primary contribution is investigating distance functions beyond Euclidean.

The tasks consisted are image classification, and language modeling.

Performance is measured in the usual way (accuracy, F1, perplexity, gradient stability). The paper also investigates interpretability, chosen to be measured the concentration of mass in a principal component decomposition of the final activations. The paper also considers sustainability, measuring grams of CO2 per run.

The paper finds that different geometry choices give different benefits, and these benefits depend on model size. No geometry wins overall. Cosine appears to work for deep vision backbones and small LMs. Euclidean for language model pretraining perplexity, and coordinate-wise distances for representation concentration.

Critical strengths

1. The paper investigates many different design choices in numerous settings.

This is a wide investigation that begins to tackle a question probably many modeling people are wondering - which is whether the loss function geometry is important - particularly for language modeling in recent years.

The range of domains, metrics, and geometries considered makes this a useful starting point for this question.

2. On a pretrained 7B model, swapping only the output head improves class structure and out-of-distribution detection at no cost in accuracy or compute.

Additionally, this swap trick presented in the paper is a cheap, drop-in change with a practical benefit, supported by repeated seeds and corrected tests across four tasks, and is worth wider investigation in the community.

Critical weaknesses

1. Hyperparameters are shared across losses rather than tuned for each, which may bias the comparison.

The learning rate and schedule are shared across all losses (Table 7). The paper argues this gives a controlled comparison. Losses with different output scales generally have different optimal learning rates: under SGD, as used for ResNet50, if L1=cL2L_1 = cL_2 then the optimal learning rate for L1L_1 is c−1c^{-1} that of L2L_2. One shared setting can therefore favour some losses over others.

For the language model setting, the paper uses AdamW with (β1,β2)=(0.9,0.999)(\beta_1, \beta_2) = (0.9, 0.999) (Appendices E and H), rather than the modern standard of (0.9,0.95)(0.9, 0.95) for transformer pretraining, used by GPT-3 [1], LLaMA [2] and SmolLM2 itself [3]; this matters because improved gradient stability relative to cross-entropy is one of the paper's headline language-model results.

Some loss functions may outperform cross-entropy simply because the cross-entropy baseline is weak: in Table 35 (CIFAR-10, ResNet50), cross-entropy gets 84.3% while Euclidean harmonic gets 91.9% and cosine 92.6%, a low number for this setting and a larger gap than between the leading distances. Comparisons against cross-entropy also take the best of many non-Euclidean variants against a single cross-entropy configuration (Section 4.2).

This is consequential because the improvements over cross-entropy are a main selling point of the paper; without per-loss tuning I cannot tell whether they reflect the loss or the baseline.

[1] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, et al. "Language Models are Few-Shot Learners." arXiv:2005.14165, 2020. https://arxiv.org/abs/2005.14165

[2] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, et al. "LLaMA: Open and Efficient Foundation Language Models." arXiv:2302.13971, 2023. https://arxiv.org/abs/2302.13971

[3] L. Ben Allal, A. Lozhkov, E. Bakouch, G. Martín Blázquez, G. Penedo, et al. "SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model." arXiv:2502.02737, 2025. https://arxiv.org/abs/2502.02737

2. The conclusions about geometry depend on the exponent, but the paper states them as general.

The paper fixes the harmonic exponent nn per setting, and implementation also varies with setting:

  • n=1n=1 with log-space logits for vision
  • n=dn=d with a probability floor for the language models
  • n=dn=\sqrt{d} for the 7B study.

Appendix E claims this makes differences "directly attributable to the geometric properties of the chosen distance" (line 1163).

The only time in the paper the exponent is varied is in Table 21, which shows that the comparison between distances itself depends strongly on nn. The paper also notes "the exponent matters more than the distance" (line 422), and vision uses n=1n=1, which Table 21 notes "cannot produce a peaked distribution and is badly calibrated". In Table 2, changing only the cosine exponent (m=16m=16 vs d\sqrt{d}) flips its comparison with cross-entropy and Euclidean at matched perplexity.

Rankings of distances therefore hold only at each setting's chosen exponent. Apart from Appendix H, which says the SmolLM2 rankings are not claimed to be invariant to nn, the paper states them as general:

  • "Bray-Curtis, cosine and Mahalanobis heads improve on Euclidean harmonic loss in most of the dataset–backbone settings we measure" (abstract)
  • "Cosine is the best all-around choice" (Section 4.2 takeaway)
  • "at 7B scale, the leading concentration gains come from Hamming, Bray-Curtis, and Manhattan" (Section 4.3 takeaway)
  • "normalized Bray-Curtis improves most consistently over Euclidean harmonic loss" (conclusion)

In vision, none of the comparisons with Euclidean survives Holm correction (Table 22).

This is consequential because the paper's central claim is that the distance is a meaningful design axis. As it stands, I cannot separate the effect of geometry from that of the exponent, the implementation or noise.

Questions for authors
  1. [Clarification] Do the vision rankings of distances (Table 22) hold if each head uses its best exponent from a small sweep, e.g. n∈{1,d,d}n \in \{1, \sqrt{d}, d\}, in at least one setting? If they do, the geometry conclusions stand. If they change, the main recommendations would need to be restated as conditional on the exponent.

  2. [Clarification] Do the improvements over cross-entropy hold with a tuned cross-entropy baseline, or with the learning rate (and, for the language models, β2\beta_2) tuned separately for each loss? For example, cross-entropy reaches 84.3% on CIFAR-10 with ResNet50 (Table 35), which is well below typical results for this setting.

  3. [Clarification] Does the out-of-distribution gain on the 7B model (Table 3) survive against a temperature-scaled cross-entropy baseline?

4 comments

Thank you for highlighting the practical value of changing the output head on the pretrained 7B model and for recognizing the evidence across tasks and seeds. We also appreciate your pointing out how the original CIFAR-10 baseline and shared training settings affected the comparison. Your suggestions have strengthened our baseline, helped us assessing the impact of the exponent and AdamW settings, and calibrate the CLINC150 baseline. We believe that the new experiments have significantly helped showcasing our work and the different nuances of harmonic losses.

Tuned vision baselines and exponents. Inspired by your feedback, we rans a shared 200-epoch ResNet50 experiment with a learning rate chosen separately for each head on validation data. Cross-entropy reached 93.74% on CIFAR-10, up from 84.3% in the original fixed-recipe comparison. Then, we tuned each harmonic head's exponent on the same validation split and evaluated the chosen configurations on three same test seeds. We repeated the full tuning procedure independently on CIFAR-100 as you requested. In summary:

HeadCIFAR-10 test accuracy (%)CIFAR-100 test accuracy (%)
Cross-entropy93.74 ± 0.2273.06 ± 1.60
Cosine-softmax93.31 ± 0.3173.72 ± 0.38
Euclidean harmonic94.37 ± 0.1074.07 ± 0.39
Cosine harmonic94.69 ± 0.0474.64 ± 0.42
Normalized Bray–Curtis harmonic94.18 ± 0.2574.12 ± 0.91

Entries are means ± SD over the three seeds. Interestingly, Cosine harmonic actually gained +0.95 ± 0.26 points over cross-entropy on CIFAR-10 and +1.59 ± 1.96 on CIFAR-100 (paired mean ± SD). Across the two datasets, cosine selected exponent 1 and Euclidean selected √d. Bray–Curtis selected 1 on CIFAR-10 and √d on CIFAR-100. In the earlier CIFAR-10 experiment using the same log-space formula and exponent grid across harmonic heads, cosine was 0.35 points ahead of Euclidean. The revised vision section and the two strong-budget appendix sections in our paper present the results and settings. We now describe the rankings in terms of the tested experiment and chosen exponents.

AdamW in SmolLM2. Thank you for pointing us to the GPT-3, LLaMA, and SmolLM2 papers, which we now cite as precedents for testing β₂ = 0.95. Thank you for suggesting this control. At β₂ = 0.95, cross-entropy achieved a mean validation perplexity of 22.09, followed closely by cosine harmonic at 22.13 and Euclidean harmonic at 22.19 across two matched seeds and 5,000 steps. Euclidean harmonic had the lowest perplexity in the submitted β₂ = 0.999 runs (21.44 versus 21.82 for cross-entropy). The change in ranking narrows our original claim and makes optimizer choice an important part of the comparison. We are following up with intermediate β₂ values under the same training recipe to see whether either harmonic head can consistently improve on cross-entropy across seeds.

Calibrated out-of-scope detection. Thank you for the interesting idea. We fitted a temperature setting for every CLINC150 head using validation data as requested. Across five seeds, calibrated cross-entropy scored 0.971 AUROC and had a 13.3% false-positive rate at 95% detection. The three harmonic heads scored 0.983–0.985 AUROC with 5.8–6.1% false positives. Tuned cosine-softmax was close at 0.983 AUROC and 6.7% false positives. The revised Language section and CLINC150 appendix now report these comparisons in detail.

As of now, we are running the above mentioned SmolLM2 experiment at intermediate AdamW settings to follow up on your optimizer question. We would be glad to discuss any concern that remains. If the revised comparisons address the issues behind your original rating, we would be grateful if you would consider raising your score.

Thank you for suggesting the AdamW control. We ran a follow-up SmolLM2 experiment at β₂ = 0.975 with the same three heads and 5,000 training steps. We were encouraged to see that Cosine harmonic had the lowest validation perplexity, averaging 21.707 versus 21.727 for cross-entropy and 21.745 for Euclidean harmonic. The gains over cross-entropy were small but consistent across seeds. Cross-entropy led at β₂ = 0.95, and Euclidean harmonic led at the submitted β₂ = 0.999.

These results motivate tuning β₂ jointly with the head. We report all three settings in Table 16 of the revised paper. Thank you for the useful comment.

Thank you also for recommending the GPT-3, LLaMA, and SmolLM2 papers. We now cite all three in the revised manuscript as precedents for AdamW β₂ = 0.95 in pretraining.

Table 16 in the SmolLM2 appendix also reports gradient stability. Across the two matched seeds, cross-entropy had the lowest mean late-training gradient-norm coefficient of variation at β₂ = 0.95 (0.0352 versus 0.0824 for cosine harmonic). Cosine harmonic had the lowest value at β₂ = 0.975 (0.0321 versus 0.0864 for cross-entropy). These results suggest that the gradient-stability ranking depends on β₂.

We would be glad to discuss any remaining concerns. If the revised comparisons are satisfactory, we would be grateful if you would consider raising your score.

Thank you for the detailed response and new experiments.

Weakness 1. Hyperparameters are shared across losses rather than tuned for each, which may bias the comparison.

Thank you for the tuned controls.

I am happy to see that in the 200-epoch control (Appendix R, Table 45) cross-entropy reaches 93.74% on CIFAR-10 (against 84.3% in the shared-recipe benchmark), much closer to what I expected. This reduces the cosine harmonic gain to +0.95 ± 0.26 points, and the CIFAR-100 gain is within seed variation (+1.59 ± 1.96, Table 46).

However, the main text still uses the weaker baseline: Table 2 has cross-entropy at 83.45%, the claim that "each harmonic head improves accuracy over equally tuned cross-entropy" rests on it, and the vision section still reports "+8.3 on CIFAR-10 with ResNet50 (0.926 against 0.843)". I would make the 200-epoch control the main-text comparison.

The SmolLM2 β2\beta_2 control (Table 16) is also helpful. I appreciate that the main text now says the perplexity ranking depends on β2\beta_2 [L411–415].

The gradient-stability ranking in Table 16 also flips with β2\beta_2, though the paper notes that at this scale seed-to-seed variation is comparable to the differences between heads [L1615–1617]. This was my original concern: the abstract still says that "for smaller models, cosine-based harmonic losses improve gradient and learning stability" [L023–024], yet every small-model run uses β2=0.999\beta_2 = 0.999 (Table 6), so that claim is untested at other settings.

Weakness 2. The conclusions about geometry depend on the exponent, but the paper states them as general.

Thank you for narrowing the claims in the main text.

I note the abstract is unchanged: "Bray-Curtis, cosine and Mahalanobis heads improve on Euclidean harmonic loss in most of the dataset–backbone settings we measure". In both tuned controls, Bray–Curtis does not improve on Euclidean on CIFAR-10 (Table 2: 92.19% vs 92.83%; Table 45: 94.18% vs 94.37%), and Mahalanobis is not included, so this sentence still depends on the shared-recipe comparisons that I found problematic. I recommend restating the claim in terms of the tuned results, as they are the more reliable comparison.

Q3. Does the out-of-distribution gain on the 7B model survive against a temperature-scaled cross-entropy baseline?

Thank you for adding the calibrated baseline (Table 23). It shows the gain does survive: calibrated cross-entropy reaches 0.971 AUROC and 13.3% FPR95, against 0.983–0.985 and 5.8–6.1% for the harmonic heads. Tuned cosine-softmax performs similarly (0.983, 6.7%), and its difference from the harmonic heads is not significant (all adjusted p≥.343p \geq .343); I am happy to see that the paper states this openly [L1780–1781].

Score

The tuned controls address much of Weakness 1, but the main-text comparisons and the abstract still rest on the weaker baselines, and the abstract's small-model stability claim is untested beyond β2=0.999\beta_2 = 0.999. I am raising Informativeness from 1 to 2: the tuned controls and the calibrated out-of-distribution comparison taught me something useful. My assessment is otherwise unchanged, and I am keeping my Rating at 2. Thank you again for the 200-epoch and temperature-scaled controls, and for your engagement during the rebuttal process.

Thank you for the thoughtful reply, and for recognizing the value of the tuned and calibrated experiments. We agree that the abstract and the main text should reflect the stronger evidence. We made the following changes, in brown.

1. Tuned vision results in main text. Results in Table 2 now reports the 200-epoch ResNet50 experiments on CIFAR-10 and CIFAR-100, including cross-entropy, cosine-softmax, and the three harmonic heads. The cross-entropy reference on CIFAR-10 is now 93.74%, and the improvement of the cosine harmonic head is +0.95 ± 0.26 percentage points. The main text also reports the improvement of +1.59 ± 1.96 points on CIFAR-100. We removed the +8.3-point comparison and the main-text significance claim, both of which were based on the earlier training schedule. We also moved the original vision radar plots to Appendix K, so that the main-text vision comparison focuses on the tuned experiments. The earlier 100-epoch experiment remains documented in Appendix V (see Table 49).

2. Gradient stability and the optimizer setting. We revised the abstract and the Language section to restrict the stability finding for the smaller models to the GPT and BERT models, trained with AdamW and β2=0.999\beta_2=0.999. We completed additional experiments with β2=0.95\beta_2=0.95, the setting used for pretraining GPT-3, LLaMA, and SmolLM2, and with β2=0.975\beta_2=0.975. These runs compare cross-entropy, the Euclidean harmonic head, and the cosine harmonic head. Table 16 reports the results, and we cite the GPT-3, LLaMA, and SmolLM2 papers that you recommended. The ranking of the heads changes with β2\beta_2. With β2=0.975\beta_2=0.975, the cosine harmonic head obtains the lowest validation perplexity (21.707, compared with 21.727 for cross-entropy and 21.745 for the Euclidean harmonic head) and the lowest gradient-norm variability (coefficient of variation 0.032, compared with 0.086 and 0.073). It also has the lowest gradient-clipping rate with both β2=0.95\beta_2=0.95 and β2=0.975\beta_2=0.975. Cross-entropy obtains the lowest perplexity with β2=0.95\beta_2=0.95, and the Euclidean harmonic head with β2=0.999\beta_2=0.999. With all three settings, cross-entropy has the highest top-1 accuracy and the Euclidean harmonic head the lowest calibration error. These comparisons motivate tuning the head and the optimizer jointly. We revised the abstract to report these results.

3. Geometry and exponent claims. In the abstract, we replaced the general ranking of Bray–Curtis, cosine, and Mahalanobis with the tuned cosine results and their variability. The vision section now states that normalized Bray–Curtis falls below Euclidean on CIFAR-10, and that Mahalanobis was evaluated only in the original benchmark. Table 2 includes the selected exponents, and Appendices R and S report the complete configurations and results. Our conclusions concern complete head configurations selected on validation data, and they remain conditional on the exponent and the training setup.

4. Benefits beyond accuracy and the calibrated OOD comparison. The tuned CIFAR-10 results also show a difference in representation structure at competitive accuracy. The cosine harmonic head requires 5.0 principal components to explain 90% of the feature variance, compared with 9.3 for Euclidean and 10.3 for cross-entropy. Its PC2 explained variance is 0.645, compared with 0.309 and 0.285, respectively. On the pretrained 7B model, harmonic heads increase class separation at similar accuracy and time per training step. The calibrated CLINC150 comparison adds a detection benefit over cross-entropy. The harmonic heads reach an AUROC of 0.983–0.985 and an FPR95 of 5.8–6.1%, compared with 0.971 and 13.3% for cross-entropy. We also state in the abstract that a tuned cosine-softmax head performs similarly on detection. This comparison does not resolve the relative contributions of normalization, the harmonic mapping, and non-Euclidean geometry. Feature concentration and retrieved examples describe representation structure. Assessing explanation faithfulness requires a separate evaluation.

We appreciate your guidance in aligning the abstract with the evidence reported in the main text. The revised presentation reports the measured benefits in predictive quality, representation structure, and detection, together with their dependence on the task, the training setup, and the computational cost.

Thank you again for your detailed feedback. We moved the tuned 200-epoch experiments to the main text, moved the original vision radar plots to the appendix, and revised the abstract to reflect the measured gains and the tested optimizer settings. We worked really hard to incorporate your feedback. If these changes address the concerns underlying your recommendation, we would be grateful if you would consider revisiting your rating. We would also appreciate clarification of any remaining issue that is decisive in your assessment.

Reviewer_KtDMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates variations of the Harmonic Loss by varying the distance measures (e.g. from Euclidean to Manhattan). The authors conduct empirical studies and measure the effect of these variations on model performance, interpretability, and sustainability. For each training task (vision and small language models), they report the winning settings.

Critical strengths

The authors conduct an extensive empirical study with a wide range of metric measures (e.g. Manhattan, Hamming, Chebyshev, Minkowski, etc.) across both vision and language tasks

Critical weaknesses

The main weakness, in my opinion, is that the paper does not give a clear takeaway. The best metric varies across settings, and I am not sure how readers should choose a metric for a new task. It would be helpful to include some theoretical analysis or justification for when and why one metric might be more suitable than another, or to discuss interesting patterns that emerge across tasks.

The remaining comments are about smaller issues, such as the lack of a standard ViT-style training setup for the vision experiments.

Questions for authors

Some suggestions:

1- Can you provide some guidance on when one distance might be more suitable than another, based on either theoretical analysis or the empirical patterns in your experiments?

2- Would you be able to you provide stronger evidence of interpretability, such as prototype visualizations or alignment with meaningful concepts, beyond concentration and clustering?

3- Does the choice of distance during training affect how well the learned representations transfer to new tasks? For example, comparing linear-probe or few-shot accuracy on new datasets could help knowing if some set-up have practical benefits beyond the original task.

1 comment

Thank you for the thoughtful review. Your questions helped us improving our experimental analysis, and make the guidance more concrete. In the following, we summarize the main updates:

1. Distance choice. We now recommend starting with cross-entropy and Euclidean harmonic as references, adding cosine when it fits the compute budget, and tuning the learning rate and harmonic exponent on validation data. The geometry gives us a reason to try a head. For example, in the attractive term of the loss, squared Euclidean pulls prototypes toward a weighted mean, whereas Manhattan pulls toward a weighted coordinate-wise median. The choice still depends on what matters for the task. In a 50-epoch ResNet50 test on SVHN, the validation rule chose cosine harmonic before test evaluation. It averaged 96.99% versus 96.68% for tuned Euclidean, a paired gain of 0.31 percentage points (95% interval −0.21 to +0.84, three seeds). We decided to add in the main text a subsection titled Practical Head Selection to discuss this aspect, and explain the SVHN test in detail in the Appendix. The original PVTv2 runs used the shared vision configuration. The new tuned vision controls use ResNet50.

2. Prototypes retrieval. Inspired by your comment, we added CLINC150 examples from six intents chosen before looking at the results. Here is one example: for credit-card APR, cosine harmonic retrieved “tell me what my credit card's apr is”, and cross-entropy retrieved “i must know my visa card's apr”. Across the six intents, 30 of 30 retrieved utterances matched the intent for each of cosine harmonic, soft Hamming harmonic, cosine-softmax, and cross-entropy. Euclidean matched 16 of 30. Although this does not imply that these examples fully explain individual predictions, cosine and soft Hamming harmonic heads retrieve utterances that closely match the intended class. It is worth noting that the Euclidean harmonic head retrieved fewer matching examples in our experiment.

We also added a quantitative test of whether training examples can approximate the classifier’s decisions. We kept each model’s feature extractor and scoring function fixed and replaced its learned class vectors with centroids of five retrieved training examples per class. On CIFAR-100, agreement with the original predictions was 99.95 ± 0.02% for cosine harmonic, compared with 85.20 ± 1.76% for cross-entropy, 75.34 ± 0.11% for Euclidean harmonic, and 75.29 ± 0.76% for cosine-softmax. These values are means ± SD across three matched seeds. Cosine harmonic also reproduced 99.91 ± 0.02% of its original incorrect predictions, showing that the approximation captures more than agreement with the correct class labels.

Random same-class centroids also achieved 99.68 ± 0.10% agreement for cosine harmonic. We therefore interpret this result as evidence that representative class examples closely approximate the cosine classifier’s decision function, while the additional benefit of selecting the highest-scoring examples is small. Appendix W, Table 50, reports the protocol and results.

We have conducted a CLINC150 pilot across five heads. On 4,500 test utterances, prediction agreement was 99.33% for cosine harmonic, 77.36% for Euclidean harmonic, 96.18% for cross-entropy, 99.02% for cosine-softmax, and 99.51% for soft Hamming harmonic. Random same-class centroids achieved 99.20% for cosine harmonic. These evaluations indicate example-based prediction fidelity.

3. Transfer to a new task. Thank you for this suggestion. We froze the ResNet50 backbones trained on CIFAR-100 and trained the same linear classifier on their features for CIFAR-10. We selected the classifier’s learning rate using CIFAR-10 validation data and evaluated the official test split across three seeds. Euclidean harmonic features achieved a test accuracy of 83.32 ± 0.36%, compared with 83.01 ± 0.72% for cross-entropy features. Cosine-softmax and normalized Bray–Curtis harmonic features reached 82.51 ± 0.26% and 82.09 ± 0.41%, respectively, while cosine harmonic features reached 53.29 ± 0.22%. All values are means ± standard deviations across the three seeds.

Euclidean's paired gain over cross-entropy was +0.31 ± 0.93 percentage points (mean ± SD). Interestingly, cosine harmonic led the end-to-end CIFAR-100 comparison, and its frozen features scored 29.22 ± 0.09 percentage points below cosine-softmax on CIFAR-10 (paired mean ± SD). Our view is that, if transfer is the goal, it would be simple to choose the most suitable head leveraging validation data from the target task. We have added the full transfer protocol and results in Appendix T, Table 47.

We hope these additions address your questions about distance selection, the relationship between prototypes and predictions, and transfer to new tasks. Thank you again for helping us strengthen the experimental analysis. If these revisions and results address your concerns, we would be grateful if you would consider raising your score.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for overseeing the discussion. We revised the paper in brown around the additional controls. KtDM valued the breadth across vision and language; Sz5x highlighted the practical value of changing the pretrained 7B model's output head; and 1JGd recognized the matched-perplexity analysis and reporting of limitations. Their questions motivated the revisions below.

1.⁠ ⁠Stronger baselines now anchor the main comparison. As requested by Sz5x, Table 2 presents the separately tuned, 200-epoch ResNet50 controls. On CIFAR-10, cross-entropy reaches 93.74% and cosine harmonic 94.69%, a paired gain of 0.95 ± 0.26 percentage points across three seeds. On CIFAR-100, cosine reaches 74.64% versus 73.06% for cross-entropy; the paired gain is 1.59 ± 1.96 points. We removed the earlier headline gain, moved the original vision radar plots to the appendix, and revised the abstract. Sz5x acknowledged that the stronger controls address much of the tuning concern.

2.⁠ ⁠Practical selection guidance tested on a new dataset. Addressing KtDM's question, we recommend including cosine alongside Euclidean harmonic and cross-entropy in a validation search over learning rates and exponents. On SVHN, our validation rule selected cosine before evaluation, and we verified this choice across three seeds. The tuned CIFAR-10 comparison also shows greater representation concentration: cosine requires 5.0 principal components for 90% of variance, versus 9.3 for Euclidean and 10.3 for cross-entropy. We describe this as representation structure, with no implication of human interpretability.

3.⁠ ⁠Calibrated detection with the relevant controls. Both Sz5x and 1JGd acknowledged that the CLINC150 detection gain remarkably survives temperature calibration of cross-entropy. Across five seeds, FPR95 is 13.3% for calibrated cross-entropy, 6.1% for Euclidean harmonic, 5.8% for cosine and soft Hamming harmonic, and 6.7% for tuned cosine-softmax. Following 1JGd's observation, the revised text explicitly attributes most of the gain to the Euclidean reference, and our results support the comparison with calibrated cross-entropy more strongly than a unique non-Euclidean advantage.

4.⁠ ⁠New replicated prediction-fidelity and intervention tests. Following 1JGd's distinction between retrieval relevance and explanation faithfulness, we replaced class vectors with centroids of five retrieved training examples. Across three seeds, cosine preserved 99.95 ± 0.02% of predictions on CIFAR-100 and 99.34 ± 0.10% on CLINC150, versus 75.34 ± 0.11% and 74.51 ± 4.27% for Euclidean. The new CLINC150 study covers 15 models and 4,500 test utterances per model. Euclidean's random control reaches 98.42 ± 0.30%. On 300 fixed utterances per model, deleting gradient-ranked words reduced cosine's calibrated confidence more than random deletion, by a paired 6.63 ± 1.29 points. Random-centroid gradients and standard baselines produced comparable effects, and retaining selected words showed limited sufficiency. These results highlight prediction fidelity and intervention sensitivity; complete causal explanation faithfulness and human usefulness remain open.

5.⁠ ⁠Transfer reveals an important limitation. At KtDM's suggestion, we tested validation-tuned linear probes on frozen CIFAR-100 features. On CIFAR-10, Euclidean reached 83.32% versus 83.01% for cross-entropy, while cosine reached 53.29%, despite leading the source-task comparison. We retain this negative result and recommend target-task validation when transfer is the objective (Appendix T).

6.⁠ ⁠Optimizer dependence is explicit. At AdamW β₂ = 0.975, cosine had the lowest SmolLM2 perplexity on both seeds after 5,000 steps: mean 21.707, versus 21.727 for cross-entropy and 21.745 for Euclidean. Cross-entropy led at 0.95 and Euclidean at 0.999. Table 16 reports all three settings; gradient-stability rankings also change. As requested by Sz5x, the abstract now limits the smaller-model stability finding to the tested β₂ = 0.999 recipe. We recommend joint head and optimizer tuning.

The revised paper provides practical guidance on predictive quality, representation structure, transfer, and cost. We hope the stronger controls and narrower claims help resolve the reviewers' concerns.

Thank you for your consideration! The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.