- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on how distance choice affects harmonic loss by extending the original Euclidean formulation to alternatives such as cosine, Manhattan, Mahalanobis, and differentiable Hamming variants. It compares predictive performance, representation structure, and training cost across vision and language tasks, including SmolLM2 pretraining and classification with a pretrained 7B model. It also examines when these alternatives improve accuracy, produce more structured representations, or offer favorable computational trade-offs.
Critical strengths
Broad empirical evaluation. The paper compares multiple distances across vision and language tasks, with additional loss baselines, repeated seeds, and frozen-backbone and exponent ablations.
Careful analysis of confounders and limitations. Matched-perplexity comparisons help separate representation concentration from predictive quality. The paper also clarifies theoretical limits and reports failed configurations and computational overhead.
Useful insights into scaling. The study shows that head rankings and computational costs change with task and output-space size, clarifying why gains in small models or classification may not transfer to vocabulary-scale language modeling.
Critical weaknesses
The explainability evidence is limited. The PCA concentration and silhouette coefficient can reflect the representation structure, but they cannot directly prove that the features or predictions are easier to be understood by people.
There are confounding factors in comparison. The differences in effective index, cosine form and numerical processing make it difficult to distinguish the independent effects of distance selection. Out-of-distribution detection also lacks a calibrated cross-entropy baseline.
The method innovation is limited. The main extension is to replace the mature distance function in the existing harmonic loss framework. The experimental research is valuable, but the guidance on when and why to choose a certain distance is still relatively limited.
The actual benefits are not stable. The visual comparison has not been corrected by multiple test corrections. The Euclidean distance achieves the best SmolLM2 perplexity, while the representation improvement of soft Hamming is accompanied by significant computational overhead.
Questions for authors
i. Under matched effective exponents and stable log-space implementations, which gains remain attributable to non-Euclidean geometry rather than exponent choice, cosine probability mappings, or numerical safeguards?
ii. Beyond matched-quality PCA and silhouette scores, what demonstrates improved interpretability? Given the weaker prototype–class-mean alignment, can the authors show a concrete task where these heads provide more faithful or useful explanations?
iii. Given the non-significant corrected vision comparisons, Euclidean’s best SmolLM2 perplexity, and Hamming’s overhead, what reproducible advantage remains over well-tuned Euclidean harmonic loss? Can a distance-selection rule independent of test results generalize to held-out settings under comparable tuning and compute budgets?
iv. Do the OOD gains persist against validation-calibrated cross-entropy and tuned cosine-softmax? What separates the contribution of non-Euclidean geometry from confidence calibration or the general benefit of harmonic heads?
2 comments
Thank you for asking how to choose a head on a new dataset. We tested our validation-based selection rule on SVHN. All three heads received the same 50-epoch training budget. We tuned each head on validation data and fixed the choice before evaluating on three fresh matched seeds. Mean test accuracy was 96.99% for cosine, 96.68% for tuned Euclidean, and 96.26% for cross-entropy. The paired gain over tuned Euclidean was 0.31 percentage points (95% interval −0.21 to +0.84). This gives a concrete example of how to use the rule, with uncertainty about the size of the gain. Table 48 in the revised paper gives the selected settings and full results. We hope this addresses your main concerns, and we would be grateful if you would reconsider your score.
Thank you for recognizing the breadth of the experiments and the value of comparing representations at matched perplexity. We also appreciate your questions about which benefits remain after tuning and what the representation measures mean in practice. We have revised the claims and added the following checks:
i) Distance, exponent, and implementation. We used the same stable log-space formula and the same validation search over learning rates and exponents for all harmonic heads. After tuning, cosine harmonic needed 5.0 principal components on average to explain 90% of the variance, compared with 9.3 for Euclidean harmonic, while achieving similar test accuracy (93.18% vs. 92.83%). Normalized Bray–Curtis also needed 5.0 components and scored 0.63 percentage points below Euclidean. This gives a practical reason to include cosine in the validation search when representation concentration matters and accuracy needs to stay close to the Euclidean reference. The result applies to the validation-tuned configurations under a common numerical implementation. We hope that this addresses your concern.
ii) Prototype examples. Thank you for the interesting suggestion. Based on your comment, we added real CLINC150 training utterances retrieved for six intents chosen before inspecting the results. For bill balance, cosine harmonic ranked “how much is the total due for my pgw gas bill” first. Euclidean ranked “what questions do you respond to” first, which belongs to another intent. Across all six intents, cosine harmonic, soft Hamming harmonic, cross-entropy, and cosine-softmax each retrieved 30 of 30 examples with the matching label. Euclidean matched 16 of 30. The appendix shows all first-ranked examples in detail. We use “representation structure” for PCA and silhouette and report the examples as a direct check of what the heads retrieve.
iii) Head transferability across datasets. On SVHN, we gave all heads the same 50-epoch budget, tuned on validation data, and froze the choice before testing on three fresh seeds. The rule chose cosine harmonic (learning rate 0.3, exponent 1). Mean test accuracies were 96.99 ± 0.16% for cosine, 96.68 ± 0.06% for Euclidean, and 96.26 ± 0.13% for cross-entropy. The selected head gained +0.31 ± 0.21 points over tuned Euclidean (paired mean ± SD, 95% interval −0.21 to +0.84). The 200-epoch CIFAR-10 and CIFAR-100 controls likewise put cosine ahead of tuned Euclidean by +0.32 ± 0.14 and +0.57 ± 0.03 points, respectively. Although the mean accuracy gains can be regarded as modest, on CIFAR-10, cosine harmonic also concentrated the representation into fewer principal components while achieving similar accuracy to tuned Euclidean. The CLINC150 examples give readers a separate way to inspect what the learned heads retrieve. When considered in combination, these results give a practical reason to include cosine in the validation search. Our guidance keeps Euclidean as a reference, uses validation accuracy to identify competitive heads, and then considers representation structure and compute cost. We include this in our paper as a practical takeaway for researchers and practitioners confirmed by several experiments.
iv) Calibrated and tuned baselines. We selected the cosine-softmax learning rates on CLINC150 validation data and fitted a temperature for every head as suggested. Across five seeds, calibrated AUROC and false-positive rate at 95% detection were 0.971 / 13.3% for cross-entropy and 0.983 / 6.7% for tuned cosine-softmax. Euclidean, cosine, and soft Hamming harmonic reached 0.983 / 6.1%, 0.984 / 5.8%, and 0.985 / 5.8%. Something noteworthy here is that the gains over cross-entropy persist after calibration. The revised Language section and appendix give the full comparison.
We are continuing the follow-up experiments and would welcome additional thoughts if any. We hope that the new experiments and our revised claims address your main concerns, and we would be grateful if you would consider raising your score in support of our paper.