- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors propose a new distance measure for the severity displacement problem. In hierarchical classification, not only is the displacement of labels/classes relevant, but also how severe the displacement of that specific class is. Given the hierarchical severity of all classes and the ground-truth label, the authors propose the WRSS metric to quantify the extent to which the predicted profile differs from the optimal profile, not only in placement but also in the severity of classes that are misplaced. The authors propose WRSS, elegantly representable by a linear rank-weighted formulation, which I find closely resembles the already proposed HOPS measure (see [W1]). On a rather small evaluation suite, the authors find WRSS to be a better alternative at capturing severity displacement than HOPS.
Critical strengths
[S1] Derivation of WRSS: The proposed metric was easy to follow during its derivation, and its final instantiation as a linear rank-weighted form is done nicely. As such, it represents an easy-to-understand metric that does not depend on hierarchical distance, such as AHD.
[S2] Real-world experiments: The evaluation using a real trained classifier on the three datasets builds a stronger empirical support for the proposed method. Still, there are some problems of fully understanding the valuation, as „LLIP“ was not (for me) clearly introduced in section 7, see [W2].
Critical weaknesses
[W1] Novelty of WRSS: As far as I can judge, the transport over ranks using the Wasserstein distance is itself an interesting angle of looking at the problem. But quickly, the authors observe that the metric can be equivalently reframed in a linear-weighted form (see Eq. (8)). Looking closely at the most related metric, HOPS, they are nearly equivalent, except that HOPS has a free parameter per class. As far as I can see, WRSS can be seen as an instantiation of HOPS, with and using signed differences instead of absolute ones. The argument for signed differences is given by the authors themselves: since z is sorted nondecreasingly, the absolute value in HOPS is no longer necessary. Thus, the proposed metric can be viewed as a certain instantiation of a general HOPS idea. The authors do not provide enough detail on this connection, nor clearly explain which changes from WRSS to HOPS are exactly driving the performance. As such, I must conclude that the improvements of WRSS over HOPS may simply reflect a misspecified version of HOPS. See [Q1].
[W2] Lack of Explanations: The paper does leave out basic definitions and derivations of concepts, such as the rank and severity problem, to raw mathematical expressions without giving at least one explicit explanation. This makes the reading cumbersome, as you are always required to mentally keep note of an example to see the differences and the way of computation. I personally find that this could be easily achieved through both visual and textual explanations. Also, I find that the authors do not do a great job of providing concrete instantiations of concepts such as the hierarchy distance, the equal-severity tiers, or the LLIP procedure. Overall, this makes the paper hard to read, and following the paper's storyline becomes difficult.
Minor points, which were only slightly influenced by the decision: [MW1] Some methods, such as „Hier-COS“, „Many recognition problems have … organized taxonomy“, „AHD, “ are missing the corresponding citations to redirect the reader to the original method. Also, I have noticed that the „Hier-Cos“ publication is missing the publication venue, CIFAR100, FVC-Arcraft and iNat19 are missing publications.
Questions for authors
[Q1] Is WRSS just an (signed) instantiation of HOPS, with ? If so, then what is the benefit of the proposed WRSS formulation in comparison to HOPS? If not, where exactly do they differ? Could you, in general, shed more light on the differences between them?
[Q2] What is meant by boundary corruption, and what would be off-path subtrees for a target y? Does this not require the target to be a more specific label in the hierarchy tree?
[Q3] In Table 2, you are comparing AHD, WRSS, and HOPS? Are you using first-k predictions, or all the predictions (i.e., top-K)? In the same sense, what hierarchy distance is used in AHD? Additionally, can you provide further insight into why AHD is strong on FGVC-Aircraft? This stems from my not understanding „the oracle is itself a hierarchy distance “.
5 comments
Thanks for your response!
Remaining Concerns of WRSS (PTQZ) I now understand that WRSS and HOPS are distinct metrics, but still closely related (Eqs 8 and 12). My previous understanding came from Eq. 7, which removes the absolute value around the prefix sums, so that the WRSS numerator equals the signed, linearly rank-weighted pointwise deviation Σᵢ(K−i+1)(ẑᵢ−zᵢ), whereas HOPS aggregates absolute deviations. Also note your counterexample should be (1,-1,0) not (1,1,0). I would still encourage revising to explicitly state closeness and to highlight the differences once more.
Still, I remain concerned about WRSS, due to the point raised by PTQZ. As stated by reviewer PTQZ, WRSS has an "equivalence to a linear rank-weighted severity objective". The authors describe newly conducted controls but do not report their results, and they acknowledge that these controls perform well when the head emphasis matches the perturbation. Providing those numbers would clearly make the point about how the new WRSS metric performs compared to other baselines.
Updated Presentation Missing Regarding the improved presentation, I appreciated the planned improvements. However, since the revised definitions, running example, and figure have not been shown, I cannot yet judge whether they resolve W2.
As these two points remain open, I keep my current score.
Thank you again for pressing us on the relationship between WRSS, HOPS, and simpler linear ranking scores. We have uploaded a revised manuscript that makes these relationships explicit and incorporates the proposed mathematical and empirical controls.
Sections 3–4 now distinguish the relevant objects directly. HOPS aggregates weighted pointwise deviations,
whereas WRSS aggregates cumulative prefix imbalance,
Figure 1 gives a concrete example in which two profiles have the same unweighted pointwise deviation but WRSS assigns different transport costs because one severity displacement persists across an additional rank boundary. The revision also states prominently that, relative to the sorted optimal profile, WRSS has an exact linear rank-weighted form. The claim is therefore not that its final computation is nonlinear, but that the transport formulation jointly fixes the feasible endpoints, normalization, tie behaviour, and budget operator.
Section 4 and Appendix A add direct and hierarchy-normalized linear, logarithmic, and geometric positional controls, together with an hNDCG-style baseline. Table 2 shows that complete-profile normalized linear severity reproduces WRSS, providing an implementation check. Under top- evaluation they differ because the positional control truncates the ranking, whereas WRSS preserves omitted severity in a tail-mass bin. The stronger logarithmic and geometric variants reach and leaf/root AUROC, compared with for unit-rank WRSS. We now use this result to delimit, rather than inflate, the claim.
Appendices D and E provide proofs of attainable-worst normalization, boundedness, the budgeted worst case, and the metric properties of the underlying prefix-sum distance. We hope these additions clarify exactly where WRSS is equivalent to a normalized linear objective, where it differs from HOPS, and what remains specific to its transport construction.
Thank you for focusing on the relationship between WRSS and HOPS. The submission did not make the algebraic distinction explicit enough. We implemented the proposed control directly and will move the derivation into the method section.
W1/Q1: Is WRSS signed HOPS with eta_i=K-i+1? No. The absolute value occurs at different levels:
D_HOPS(zhat,z)=sum_i eta_i|zhat_i-z_i|
D_WRSS(zhat,z)=sum_j|sum_{i<=j}(zhat_i-z_i)|.
Sortedness makes cumulative imbalances nonnegative; it does not give every pointwise deviation the same sign. For z=(0,1,2) and zhat=(1,0,2), pointwise deviations are (1,1,0), while cumulative deviations are (1,0,0). The numerators already differ, so no common denominator can make the operators identical.
WRSS's linear reduction holds for complete target-conditioned severity profiles with equal total mass on a uniformly spaced one-dimensional rank line. Under those assumptions, summing cumulative imbalance produces transport-derived linear rank coefficients. This does not identify WRSS with arbitrary pointwise linear absolute deviation.
We implemented the proposed alternative exactly: pointwise HOPS with linear rank weights, a full reversed reference, and no clipping. It differs from WRSS on CIFAR-100, FGVC-Aircraft, and iNat19. On iNat19 at d=5, median response is 0.0016 for WRSS and 0.0159 for linear HOPS under local jitter around a damaged head, but 1.631 and 1.379 under cross-tier destruction. Their ordering reverses across perturbation semantics, excluding identity or constant rescaling. On trained CIFAR-100 models, WRSS also has lower full-ranking typical and upper-tail input-noise response for both cross-entropy and HierCOS. The algebra disproves equivalence; the experiments show that the distinction matters numerically. WRSS agrees only with its own normalized transport-derived linear form, as required.
W2: insufficient explanation and examples. The revision will use one running hierarchy example to show the target-conditioned severity profile, hierarchy-optimal profile, attainable reverse, cumulative prefix imbalance, and mass-preserving truncation:
T_k(x)=(x_1,...,x_k,sum_{i>k}x_i).
The example will contrast an optimal ranking, a ranking that exhausts HOPS profile credit, and a more severely displaced ranking that HOPS may also clip to zero but WRSS still distinguishes. This connects the algebra directly to the diagnostic use.
Minor weakness: citations. We will add direct citations and publication details at first mention for Hier-COS, AHD, CIFAR-100, FGVC-Aircraft, and iNaturalist, and strengthen the opening hierarchical-recognition references.
Q2: boundary corruption and off-path subtrees. For target leaf y, the root-to-y path selects one child at each ancestor. An off-path subtree is rooted at a sibling child that does not contain y. At a controlled path depth, boundary corruption selects such a subtree and transfers probability mass alpha uniformly to its descendant leaves. The associated oracle costs are
C_root=alpha*D_root and C_leaf=alpha*mean(D_leaf),
where D_root is the target-to-subtree-root distance and mean(D_leaf) is the mean target-to-descendant-leaf distance. We will add a tree diagram and pseudocode.
Q3: budget, distance, and AHD on FGVC-Aircraft. Every table will state whether it uses first-k, hierarchy radius d, or full ranking, and will identify AHD's hierarchy distance. AHD is strong in the near/far experiment because that oracle is constructed from hierarchy distance, especially on the shallow FGVC-Aircraft hierarchy. This is direct oracle alignment, not a setting where WRSS should dominate a distance-only target. At full ranking, AHD is constant across models because every complete ranking contains the same severity multiset; WRSS still evaluates its placement.
Additional positional controls. We tested linear, logarithmic DCG-style, and geometric RBP-style severity scores. These controls weight ranked severities pointwise and normalize between ideal and reversed-prefix costs. They are not cumulative transport and do not preserve omitted tail mass. They test whether an external positional discount suffices once target-conditioned severity and feasible endpoints are supplied.
Revision. The method will define severity profiles, feasible endpoints, and budgets before WRSS, then immediately provide the HOPS/WRSS derivation. One hierarchy figure will also define equal tiers, off-path subtrees, and boundary corruption. About half the extra page is reserved for definitions and protocol; complete positional, reference, and clipping controls remain in the appendix.
The central equivalence claim is contradicted by the operators, the minimal counterexample, the direct proposed control, and trained-model results. The revised distinction is precise: HOPS measures pointwise profile satisfaction; WRSS measures cumulative severity displacement.