acceptodds
Under review as a conference paper at ICLR 2027

WRSS: Severity Transport for Hierarchical Ranking Evaluation

Abstract

Hierarchical classification errors differ not only in correctness, but also in severity. For ranked predictions, evaluation must additionally account for where severe alternatives appear. We study this problem as severity displacement: the movement of hierarchy-derived severity toward the head of a predicted ranking. We introduce the Wasserstein Rank–Severity Score (WRSS), which measures this displacement through one-dimensional transport over rank positions. The resulting distance is equivalent to a linear rank-weighted severity objective, while the transport construction jointly determines its weighting, normalization, equal-tier invariance, and mass-preserving top- truncation. Controlled experiments separately test stability under stochastic ranking ambiguity and responsiveness to hierarchy-derived severity corruption. WRSS exhibits lower local volatility than HOPS in the tested ambiguity regimes and stronger alignment with controlled severity displacement in several fixed-budget settings. Experiments on trained classifiers show similar stability under input perturbations. WRSS therefore complements distance-based and preference-based metrics by measuring cumulative severity displacement along the ranking.

Primary area:
datasets and benchmarks
Keywords:
hierarchical classification, evaluation metrics, learning to rank, optimal transport, cost-sensitive evaluation, model assessment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XcuWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a new distance measure for the severity displacement problem. In hierarchical classification, not only is the displacement of labels/classes relevant, but also how severe the displacement of that specific class is. Given the hierarchical severity of all classes and the ground-truth label, the authors propose the WRSS metric to quantify the extent to which the predicted profile differs from the optimal profile, not only in placement but also in the severity of classes that are misplaced. The authors propose WRSS, elegantly representable by a linear rank-weighted formulation, which I find closely resembles the already proposed HOPS measure (see [W1]). On a rather small evaluation suite, the authors find WRSS to be a better alternative at capturing severity displacement than HOPS.

Critical strengths

[S1] Derivation of WRSS: The proposed metric was easy to follow during its derivation, and its final instantiation as a linear rank-weighted form is done nicely. As such, it represents an easy-to-understand metric that does not depend on hierarchical distance, such as AHD.

[S2] Real-world experiments: The evaluation using a real trained classifier on the three datasets builds a stronger empirical support for the proposed method. Still, there are some problems of fully understanding the valuation, as „LLIP“ was not (for me) clearly introduced in section 7, see [W2].

Critical weaknesses

[W1] Novelty of WRSS: As far as I can judge, the transport over ranks using the Wasserstein distance is itself an interesting angle of looking at the problem. But quickly, the authors observe that the metric can be equivalently reframed in a linear-weighted form (see Eq. (8)). Looking closely at the most related metric, HOPS, they are nearly equivalent, except that HOPS has a free parameter η\eta per class. As far as I can see, WRSS can be seen as an instantiation of HOPS, with ηi=K−i+1\eta_i = K-i+1 and using signed differences instead of absolute ones. The argument for signed differences is given by the authors themselves: since z is sorted nondecreasingly, the absolute value in HOPS is no longer necessary. Thus, the proposed metric can be viewed as a certain instantiation of a general HOPS idea. The authors do not provide enough detail on this connection, nor clearly explain which changes from WRSS to HOPS are exactly driving the performance. As such, I must conclude that the improvements of WRSS over HOPS may simply reflect a misspecified version of HOPS. See [Q1].

[W2] Lack of Explanations: The paper does leave out basic definitions and derivations of concepts, such as the rank and severity problem, to raw mathematical expressions without giving at least one explicit explanation. This makes the reading cumbersome, as you are always required to mentally keep note of an example to see the differences and the way of computation. I personally find that this could be easily achieved through both visual and textual explanations. Also, I find that the authors do not do a great job of providing concrete instantiations of concepts such as the hierarchy distance, the equal-severity tiers, or the LLIP procedure. Overall, this makes the paper hard to read, and following the paper's storyline becomes difficult.

Minor points, which were only slightly influenced by the decision: [MW1] Some methods, such as „Hier-COS“, „Many recognition problems have … organized taxonomy“, „AHD, “ are missing the corresponding citations to redirect the reader to the original method. Also, I have noticed that the „Hier-Cos“ publication is missing the publication venue, CIFAR100, FVC-Arcraft and iNat19 are missing publications.

Questions for authors

[Q1] Is WRSS just an (signed) instantiation of HOPS, with ηi=K−i+1\eta_i = K-i+1? If so, then what is the benefit of the proposed WRSS formulation in comparison to HOPS? If not, where exactly do they differ? Could you, in general, shed more light on the differences between them?

[Q2] What is meant by boundary corruption, and what would be off-path subtrees for a target y? Does this not require the target to be a more specific label in the hierarchy tree?

[Q3] In Table 2, you are comparing AHD, WRSS, and HOPS? Are you using first-k predictions, or all the predictions (i.e., top-K)? In the same sense, what hierarchy distance is used in AHD? Additionally, can you provide further insight into why AHD is strong on FGVC-Aircraft? This stems from my not understanding „the oracle is itself a hierarchy distance “.

5 comments

Thank you for focusing on the relationship between WRSS and HOPS. The submission did not make the algebraic distinction explicit enough. We implemented the proposed control directly and will move the derivation into the method section.

W1/Q1: Is WRSS signed HOPS with eta_i=K-i+1? No. The absolute value occurs at different levels:

D_HOPS(zhat,z)=sum_i eta_i|zhat_i-z_i|

D_WRSS(zhat,z)=sum_j|sum_{i<=j}(zhat_i-z_i)|.

Sortedness makes cumulative imbalances nonnegative; it does not give every pointwise deviation the same sign. For z=(0,1,2) and zhat=(1,0,2), pointwise deviations are (1,1,0), while cumulative deviations are (1,0,0). The numerators already differ, so no common denominator can make the operators identical.

WRSS's linear reduction holds for complete target-conditioned severity profiles with equal total mass on a uniformly spaced one-dimensional rank line. Under those assumptions, summing cumulative imbalance produces transport-derived linear rank coefficients. This does not identify WRSS with arbitrary pointwise linear absolute deviation.

We implemented the proposed alternative exactly: pointwise HOPS with linear rank weights, a full reversed reference, and no clipping. It differs from WRSS on CIFAR-100, FGVC-Aircraft, and iNat19. On iNat19 at d=5, median response is 0.0016 for WRSS and 0.0159 for linear HOPS under local jitter around a damaged head, but 1.631 and 1.379 under cross-tier destruction. Their ordering reverses across perturbation semantics, excluding identity or constant rescaling. On trained CIFAR-100 models, WRSS also has lower full-ranking typical and upper-tail input-noise response for both cross-entropy and HierCOS. The algebra disproves equivalence; the experiments show that the distinction matters numerically. WRSS agrees only with its own normalized transport-derived linear form, as required.

W2: insufficient explanation and examples. The revision will use one running hierarchy example to show the target-conditioned severity profile, hierarchy-optimal profile, attainable reverse, cumulative prefix imbalance, and mass-preserving truncation:

T_k(x)=(x_1,...,x_k,sum_{i>k}x_i).

The example will contrast an optimal ranking, a ranking that exhausts HOPS profile credit, and a more severely displaced ranking that HOPS may also clip to zero but WRSS still distinguishes. This connects the algebra directly to the diagnostic use.

Minor weakness: citations. We will add direct citations and publication details at first mention for Hier-COS, AHD, CIFAR-100, FGVC-Aircraft, and iNaturalist, and strengthen the opening hierarchical-recognition references.

Q2: boundary corruption and off-path subtrees. For target leaf y, the root-to-y path selects one child at each ancestor. An off-path subtree is rooted at a sibling child that does not contain y. At a controlled path depth, boundary corruption selects such a subtree and transfers probability mass alpha uniformly to its descendant leaves. The associated oracle costs are

C_root=alpha*D_root and C_leaf=alpha*mean(D_leaf),

where D_root is the target-to-subtree-root distance and mean(D_leaf) is the mean target-to-descendant-leaf distance. We will add a tree diagram and pseudocode.

Q3: budget, distance, and AHD on FGVC-Aircraft. Every table will state whether it uses first-k, hierarchy radius d, or full ranking, and will identify AHD's hierarchy distance. AHD is strong in the near/far experiment because that oracle is constructed from hierarchy distance, especially on the shallow FGVC-Aircraft hierarchy. This is direct oracle alignment, not a setting where WRSS should dominate a distance-only target. At full ranking, AHD is constant across models because every complete ranking contains the same severity multiset; WRSS still evaluates its placement.

Additional positional controls. We tested linear, logarithmic DCG-style, and geometric RBP-style severity scores. These controls weight ranked severities pointwise and normalize between ideal and reversed-prefix costs. They are not cumulative transport and do not preserve omitted tail mass. They test whether an external positional discount suffices once target-conditioned severity and feasible endpoints are supplied.

Revision. The method will define severity profiles, feasible endpoints, and budgets before WRSS, then immediately provide the HOPS/WRSS derivation. One hierarchy figure will also define equal tiers, off-path subtrees, and boundary corruption. About half the extra page is reserved for definitions and protocol; complete positional, reference, and clipping controls remain in the appendix.

The central equivalence claim is contradicted by the operators, the minimal counterexample, the direct proposed control, and trained-model results. The revised distinction is precise: HOPS measures pointwise profile satisfaction; WRSS measures cumulative severity displacement.

Thanks for your response!

Remaining Concerns of WRSS (PTQZ) I now understand that WRSS and HOPS are distinct metrics, but still closely related (Eqs 8 and 12). My previous understanding came from Eq. 7, which removes the absolute value around the prefix sums, so that the WRSS numerator equals the signed, linearly rank-weighted pointwise deviation Σᵢ(K−i+1)(ẑᵢ−zᵢ), whereas HOPS aggregates absolute deviations. Also note your counterexample should be (1,-1,0) not (1,1,0). I would still encourage revising to explicitly state closeness and to highlight the differences once more.

Still, I remain concerned about WRSS, due to the point raised by PTQZ. As stated by reviewer PTQZ, WRSS has an "equivalence to a linear rank-weighted severity objective". The authors describe newly conducted controls but do not report their results, and they acknowledge that these controls perform well when the head emphasis matches the perturbation. Providing those numbers would clearly make the point about how the new WRSS metric performs compared to other baselines.

Updated Presentation Missing Regarding the improved presentation, I appreciated the planned improvements. However, since the revised definitions, running example, and figure have not been shown, I cannot yet judge whether they resolve W2.

As these two points remain open, I keep my current score.

Thank you again for pressing us on the relationship between WRSS, HOPS, and simpler linear ranking scores. We have uploaded a revised manuscript that makes these relationships explicit and incorporates the proposed mathematical and empirical controls.

Sections 3–4 now distinguish the relevant objects directly. HOPS aggregates weighted pointwise deviations,

∑iηi∣z^i−zi∣,\sum_i \eta_i \lvert \hat z_i-z_i\rvert,

whereas WRSS aggregates cumulative prefix imbalance,

∑j∣∑i≤j(z^i−zi)∣.\sum_j \left\lvert \sum_{i\leq j}(\hat z_i-z_i)\right\rvert.

Figure 1 gives a concrete example in which two profiles have the same unweighted pointwise deviation but WRSS assigns different transport costs because one severity displacement persists across an additional rank boundary. The revision also states prominently that, relative to the sorted optimal profile, WRSS has an exact linear rank-weighted form. The claim is therefore not that its final computation is nonlinear, but that the transport formulation jointly fixes the feasible endpoints, normalization, tie behaviour, and budget operator.

Section 4 and Appendix A add direct and hierarchy-normalized linear, logarithmic, and geometric positional controls, together with an hNDCG-style baseline. Table 2 shows that complete-profile normalized linear severity reproduces WRSS, providing an implementation check. Under top-kk evaluation they differ because the positional control truncates the ranking, whereas WRSS preserves omitted severity in a tail-mass bin. The stronger logarithmic and geometric variants reach 0.777/0.6850.777/0.685 and 0.779/0.6930.779/0.693 leaf/root AUROC, compared with 0.764/0.6680.764/0.668 for unit-rank WRSS. We now use this result to delimit, rather than inflate, the claim.

Appendices D and E provide proofs of attainable-worst normalization, boundedness, the budgeted worst case, and the metric properties of the underlying prefix-sum distance. We hope these additions clarify exactly where WRSS is equivalent to a normalized linear objective, where it differs from HOPS, and what remains specific to its transport construction.

Reviewer_8ZRyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a metric to measure how far a ranking is from a hierarchy ideal ranking given a true label. The core idea is to define an ideal ranking profile based on the true label and compare it against a given ranking. A Wasserstein distance metric is used to define how far the one profile is from the other and is the main technical contribution of the paper. A few variants and some basic properties of the metric are studied. Finally, the experiments try to validate the proposed method with some experiments.

Critical strengths
  1. Clear framing of the problem. Compared to other approaches such as AHDS and HOPS, the distinction is made clear. AHDS is ranking invariant, while HOPS is not invariant, the authors show that their metric is better.

  2. The proposed score is easy to compute.

  3. The paper tests both stability and responsiveness results and notes that just one of them is insufficient.

Critical weaknesses
  1. Limited novelty and also potentially limited applications to real world problems. It simply boils down to defining a metric over two ranked profiles. The authors should elaborate in more detail on where this is likely to be useful.

  2. Since the worst case score for WRSS is based on the truly worst case ranking but for HOPS it is based on the top-K worst case, how much of this was a factor in the sensitivity results? Please see the first comment in other comments.

  3. While a fancy metric has been proposed, why not simply have a correlation between the severity and the position? Intuitively as the position becomes higher (i.e., worse ranked positions), the severity should become higher. Why not a simple metric such as the correlation be used instead of the metric proposed? If nothing else, some such simple baselines should have been compared against to strengthen the paper as to why we need the proposed metric. It is not clear why specifically a Wasserstein metric was chosen while the problem could have been addressed in other ways.

  4. Poorly written paper that leaves so many questions open to a reader (please see the other comments).

Questions for authors
  1. Is equation 14 really the worst case ranking? Wouldn't that involving resorting all z's rather than only those in the top K? Is this a fundamental limitation from the original paper that proposed HOPS? Please clarify. If it is a limitation in that paper, I think this paper should consider both HOPS and a variant of HOPS where it is based on the truly worst case ranking.

  2. LLIP is never defined in the paper from what I can tell. It is used heavily in Section 8.3.

  3. I did not fully understand how the AUC ROC metric was computed in Section 8.2 for Table 2. This requires more explanation of how this was done.

  4. The samplers in the experiments are not described in adequate detail. While some intuitive explanation has been given for each of the sampler, how were they implemented?

  5. On lines 370 to 375, what are capital Z's since they are not defined ever before in the paper. Also, what is \Delta_{log_10}? Is this a definition of this quantity from Z's? Further what is S?

  6. In Table 1, what is IQR? First time Inter Quartile Range is mentioned in the text, please use this abbreviation.

  7. In Table 1, what are the three regimes HC, TS and WT? Where have these been defined?

  8. Why is AHD performing better in Table 2?

Minor: 339 --> distinguish distnguish

3 comments

Thank you for the normalization, correlation, and reproducibility questions. They exposed genuine omissions. We now separate each metric's evaluation object and reserve revision space for definitions rather than results alone.

W1/W2/Q1: scope and HOPS normalization. WRSS measures cumulative target-conditioned severity displacement: after an error, do high-ranked alternatives remain near the target, or does severe mass enter the head? It complements AHD, HOPS, and global rank-severity agreement.

The HOPS comparison is materially affected by its budget-specific reference and clipping. We now separate published HOPS, raw HOPS, full-reference HOPS, linear full-reference HOPS, and WRSS. On controlled CIFAR-100, 99.2% of published HOPS values clip at k=5. On iNat19, 99.5% are out of range at k=10, and clipping changes median tier-destruction response from 5.805 to 0.000. On natural CIFAR-100 outputs at k=5, clipping affects 99.94% of cross-entropy and 98.42% of HierCOS examples. This is exhausted profile credit, not invariance. Full reversal repairs much of this behavior, but is not generally extremal for pointwise deviation; reversal is extremal for WRSS's cumulative one-dimensional transport. We therefore interpret HOPS as structured profile satisfaction, while WRSS continues grading displacement after profile failure.

W3: correlation and positional controls. We added full-ranking OLS. Because complete ideal and predicted profiles share the same severity multiset, its fitted slope equals Pearson correlation and is bounded in [-1,1]. Truncation is not neutral: prefixes (0,1,1,1,2) and (3,4,4,4,5) have identical perfect correlation although every retained severity differs by three. Truncation removes the shared-multiset guarantee, may unbound the slope, and can make equal-tier prefixes undefined. We therefore report OLS only at full ranking.

We also add pointwise linear, logarithmic DCG-style, and geometric RBP-style severity controls normalized between ideal and reversed-prefix costs. They are not cumulative transport and do not use WRSS's tail-mass bin.

Perturbation meaning. Low response is desirable for within-tier permutations and small local perturbations; high or monotone response is desirable when cross-tier structure or boundaries are deliberately destroyed. The revision will state each intended direction.

For trained-model noise, independent Gaussian noise is added in the original 8-bit pixel domain, clipped to [0,255], followed by the standard transform and fresh inference. WRSS is less volatile than corrected HOPS from k=20 onward; at full ranking, P90 typical adjacent response is about 40% lower for cross-entropy and 37% lower for HierCOS.

Q2: local response and LLIP. For image i and adjacent levels,

s_ij=|M_i(sigma_{j+1})-M_i(sigma_j)|/(sigma_{j+1}-sigma_j).

Typical response is median_j s_ij; we report its median, IQR, and P90 across images. LLIP_i=log10(max_j s_ij+1e-12) records worst adjacent response. Dataset entries are means of seed-level summaries; variability is the standard deviation across seeds.

Q3: AUROC. At fixed transferred mass alpha, near/far cases are lower/upper quantiles of hierarchy-derived transport cost, with far positive. Quality scores such as WRSS/HOPS are negated; costs such as AHD retain their sign. AUROC uses the Mann-Whitney statistic and average ranks for ties. Conditions missing either class are undefined; we never use max(AUROC,1-AUROC).

Q4: samplers. Tier-softmax constructs hierarchy-plausible logits and adds configured Gaussian/Gumbel noise. Within-tier shuffle permutes only equal-severity blocks. Head corruption swaps selected head logits with far-severity labels. Boundary corruption transfers mass alpha into an off-path subtree; oracle costs are alphaD_root and alphamean(D_leaf). Controlled logit noise perturbs synthetic logits; trained-model input noise perturbs 8-bit pixels and reruns inference. Exact configurations and pseudocode will be added to the appendix.

Q5-Q7. We will define every symbol before use, expand IQR/HC/TS/WT, state metric direction, k/d/full interface and aggregation unit in every caption, and restrict range diagnostics to metrics intended for [0,1].

Q8. AHD can outperform WRSS when the oracle is hierarchy distance because AHD is directly aligned with it. AHD measures average retained severity; HOPS profile satisfaction; OLS global order; WRSS cumulative displacement. Agreement is desirable; disagreement identifies different failure semantics.

The extra page will prioritize these definitions and the normalization comparison; full pseudocode, defaults, controls, and grids remain in the appendix.

Thank you again for requesting clearer experimental definitions, stronger comparisons, and a more qualified account of the stability results. We have uploaded a revised manuscript that expands the protocol, adds the requested controls, and narrows the conclusions to what the experiments establish.

The controlled samplers and their roles are now defined in Appendix F: tier-softmax tests ambiguity around a hierarchy-consistent ranking, within-tier sampling tests exchanges among equal-severity labels, and head corruption moves distant-severity labels into early ranks. The external severity references are defined in Appendix G, including the distinction between root- and leaf-level costs and the known displaced-mass coefficient used only in controlled corruption. Appendix H now gives the complete finite-difference aggregation, perturbation grids, raw-pixel noise procedure, and the distinction between the controlled local-sensitivity statistic and the trained-model LLIP proxy.

The main results are correspondingly more qualified. In Table 1, WRSS has lower median and 9090th-percentile sensitivity than HOPS in all three controlled regimes, with median log-scale differences of −0.92-0.92, −0.76-0.76, and −1.49-1.49. Table 3 separately tests responsiveness: WRSS exceeds HOPS for all three leaf-level oracles and for two of three root-level oracles, while AHD remains strongest in several distance-derived settings. Table 4 reports the important exceptions across budgets, including negative WRSS--HOPS differences for FGVC-Aircraft at some intermediate cutoffs.

Table 5 then shows lower aggregate WRSS input-jitter sensitivity across every displayed dataset, training objective, and budget regime, with original-scale sensitivity ratios from 0.0560.056 to 0.5440.544. The accompanying text also reports the exception that HOPS is slightly less sensitive at k=5k=5 on CIFAR-100.

We therefore no longer infer usefulness from stability alone or claim uniform dominance. The revision separates stability, controlled severity responsiveness, and trained-model diagnosis, and explicitly treats the natural-error analysis as metric characterisation under distance-derived references.

I thank the authors for their response. I decide to keep my score as the novelty and the practical utility of the method were both unclear after reading the rebuttal.

Reviewer_PTQZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WRSS (Wasserstein Rank-Severity Score), a hierarchical ranking metric that measures severity displacement: not only how severe prediction errors are, but how prominently severe errors appear in the ranked output.

Its main contribution is a principled 1-D optimal-transport formulation that yields rank-sensitive severity weighting, normalization, equal-severity invariance, and mass-preserving top-k evaluation, complementing AHD’s distance-based and HOPS’s preference-based views.

Validation on CIFAR-100, FGVC-Aircraft, and iNat19, using controlled corruptions and trained classifiers, shows that WRSS is consistently more stable than HOPS while generally providing stronger discrimination of near versus far hierarchical errors, although AHD remains stronger when evaluation targets are explicitly distance-based.

Critical strengths
  1. Clear and meaningful evaluation objective. The paper identifies a useful gap in hierarchical ranking evaluation: existing metrics capture hierarchical error severity (AHD) or preference consistency (HOPS), but do not directly quantify how much severe error is displaced toward the top of a ranked prediction. WRSS provides a principled formulation of this “severity displacement” through 1-D optimal transport, with desirable properties such as rank sensitivity, normalization, equal-severity invariance, and coherent top-k handling. This is consequential because it adds a distinct and interpretable evaluation dimension rather than simply proposing another variant of an existing metric.

  2. Well-designed validation demonstrating both stability and severity sensitivity. The experiments use controlled perturbations as well as trained classifiers across CIFAR-100, FGVC-Aircraft, and iNat19, and explicitly test both robustness to irrelevant/local ranking variation and responsiveness to genuinely more severe errors. WRSS is consistently more stable than HOPS while showing stronger near-vs-far severity discrimination in most tested settings. This combination provides meaningful empirical evidence that WRSS captures its intended notion of severity displacement rather than obtaining improvements simply by being less sensitive to ranking changes.

Critical weaknesses
  1. The methodological novelty is somewhat limited by WRSS’s equivalence to a linear rank-weighted severity objective. Although the optimal-transport formulation provides an elegant interpretation and derives several desirable properties, the resulting metric can ultimately be expressed as a normalized linear weighting of hierarchical severity by rank. This raises the question of whether Wasserstein transport provides a substantively new evaluation framework or primarily a principled derivation of a relatively simple weighted metric. The paper would be stronger with comparisons against carefully designed rank-weighted severity baselines, including alternative discount functions, to demonstrate that the specific WRSS construction provides benefits beyond this simpler family.

  2. The empirical validation does not yet fully establish the practical advantage of WRSS on natural model errors. Much of the clearest evidence comes from controlled corruption experiments constructed to test severity displacement, which is precisely the property WRSS is designed to capture. Results on trained classifiers and natural errors are more mixed, and AHD can outperform WRSS when the evaluation target is explicitly hierarchy-distance based. Consequently, while the experiments convincingly demonstrate that WRSS behaves as intended, they provide weaker evidence that severity displacement is the preferable evaluation criterion in realistic downstream settings or that WRSS would materially change model selection and conclusions in practice.

Questions for authors
  1. Clarification: Since WRSS is shown to be equivalent to a normalized linear rank-weighted severity objective, what empirical or theoretical advantage does the Wasserstein formulation provide over simpler rank-weighted severity baselines? In particular, have you compared WRSS against otherwise identical metrics using linear or standard ranking discounts? A convincing demonstration that the transport formulation provides benefits beyond a principled reinterpretation of weighted severity would materially strengthen my assessment of the paper’s novelty.

  2. Clarification: The controlled corruption experiments demonstrate that WRSS behaves as intended, but the results on natural model errors appear less decisive. Can you provide evidence that WRSS leads to meaningfully different or better conclusions in a realistic model-selection setting, for example by identifying cases where WRSS correctly distinguishes two trained models or checkpoints that AHD/HOPS would rank incorrectly according to a relevant downstream criterion? Such evidence would materially affect my assessment of the practical significance of severity displacement.

Suggestion for future work, not required for the assessment: It would be interesting to study alternative rank geometries or discount functions within the transport framework and characterize when linear rank weighting is preferable to more strongly head-weighted alternatives.

2 comments

Thank you for identifying the two questions central to the contribution: whether WRSS adds anything beyond a positional discount, and whether the distinction is useful on trained models. The new controls sharpen our claim: WRSS is a bounded, budget-aware diagnostic of cumulative target-conditioned severity displacement.

W1/Q1: novelty beyond linear/logarithmic/geometric scores. For complete equal-mass profiles on a uniform one-dimensional rank line, WRSS has an exact normalized linear form. We will state this prominently. The contribution is not nonlinear computation, but the transport-derived construction: target-conditioned severity, hierarchy-optimal and attainable-worst profiles, equal-tier handling, and mass-preserving truncation. These determine the coefficients and normalized range rather than selecting a discount post hoc.

We added pointwise positional controls

C_w(pi;y) = sum_{i=1}^k w_i r_y(pi_i),

S_w = 1 - [C_w(pi;y)-C_w(pi*;y)]/[C_w(pi_rev;y)-C_w(pi*;y)]

with linear, logarithmic w_i=1/log2(i+1), and geometric w_i=(1-p)p^(i-1) weights. These are normalized pointwise severity controls, not cumulative transport or standard DCG/RBP used unchanged. Standard hNDCG supplies ideal normalization after converting severity to gain, but does not map the attainable worst ranking to zero without an added feasible-range construction. Our controls add that construction. They also truncate directly, whereas WRSS preserves omitted severity mass in a tail bin. Their strong performance when head emphasis matches the perturbation is informative: external rank utility can be added once the evaluation object and feasible range are defined. Uniform WRSS is the untuned baseline when no external positional preference is justified.

We also implemented the proposed linear-HOPS control with full reversal and no clipping. It is not WRSS:

D_HOPS = sum_i eta_i |zhat_i-z_i|

D_WRSS = sum_j |sum_{i<=j}(zhat_i-z_i)|.

For z=(0,1,2) and zhat=(1,0,2), pointwise deviations are (1,1,0), while cumulative deviations are (1,0,0); normalization cannot make these numerators identical. Controls on CIFAR-100, FGVC-Aircraft, and iNat19 confirm the distinction. WRSS agrees only with its own transport-derived linear form.

W2/Q2: practical significance. The submitted natural-error averages did not make the use case concrete. WRSS is primarily diagnostic: after top-1 prediction, how much target-conditioned severity enters the visible head? We do not claim that one hierarchy score universally determines model selection.

On CIFAR-100, cross-entropy and HierCOS differ by only 0.084 accuracy points, while global hierarchy-aware metrics broadly agree that HierCOS is slightly better. This agreement is desirable. The question is whether truncation preserves finer information after the aggregate conclusion is known.

At k=5, published HOPS ties the models for 90/100 target classes; its class-wise difference has Q25=median=Q75=0. Median class-seed clipping is 100% for both methods; every cross-entropy class-seed group and 98% of HierCOS groups exceed 90% clipping. "Clipped" means the raw score lies outside its interval and is changed by the clipping rule.

WRSS ties only 3 classes and retains an IQR of [-0.0353,0.0292]. The same method is preferred across all five seeds for 32 classes under WRSS, versus 2 under HOPS. At k=20, WRSS resolves 97 comparisons versus 67, with all-seed consistency for 39 versus 17. WRSS therefore recovers target-specific gains and losses erased by the published HOPS reference and clipping without manufacturing a contrary global ordering. Full-reference unclipped HOPS recovers much of this resolution, identifying the cause, but is a modified object and is not generally bounded across datasets.

We also tested whether the extra resolution is stable. Models were rerun under matched Gaussian noise in the original 8-bit pixel domain, clipped to [0,255] before evaluation. For adjacent noise levels,

s_ij(M)=|M_i(sigma_{j+1})-M_i(sigma_j)|/(sigma_{j+1}-sigma_j).

Typical response uses median_j s_ij; LLIP uses log10(max_j s_ij+1e-12). Entries are means of seed-level summaries. WRSS is less volatile than corrected HOPS from k=20 onward. At full ranking, typical response is about 32% lower for cross-entropy and 20% lower for HierCOS; P90 is about 40% and 37% lower. Linear full-reference HOPS remains more volatile.

Thus WRSS's value is stable class- and budget-specific diagnosis when aggregate metrics agree and published HOPS has exhausted profile credit. The revision will prioritize definitions, the pointwise/cumulative distinction, and one class-conditioned trained-model table; full grids and pseudocode remain in the appendix.

Thank you again for raising the central questions of novelty beyond positional weighting and practical value on trained models. We have uploaded a substantially revised manuscript that incorporates the proposed changes and additional analyses described in our response.

The conceptual distinction is now stated directly in Sections 1–3 and illustrated in Figure 1. For complete equal-mass profiles on a uniformly spaced rank line, WRSS admits an exact normalized linear form; we therefore no longer present nonlinearity of the final computation as the contribution. Instead, the transport construction jointly determines the target-conditioned severity profile, hierarchy-optimal and attainable-worst endpoints, feasible-range normalization, equal-tier invariance, and mass-preserving top-kk operator.

Section 4 and Appendix A introduce matched linear, logarithmic, geometric, and hNDCG-style controls to separate these components. Table 2 shows that direct positional sums remain at chance AUROC, whereas hierarchy-normalized variants recover the controlled oracle. Stronger logarithmic and geometric discounts can outperform unit-rank WRSS when matched to the corruption, which we now acknowledge explicitly rather than claiming universal optimality.

The practical-value analysis is expanded in Section 5.5. On FGVC-Aircraft, Hier-COS improves accuracy by 2.212.21 points, while its WRSS advantage grows from 0.01840.0184 at k=1k=1 to 0.16120.1612 at k=20k=20, corresponding to a 42.4%42.4\% reduction in severity displacement. Table 7 further shows that WRSS resolves 97/10097/100 class comparisons at both k=5k=5 and k=20k=20, compared with 10/10010/100 and 67/10067/100 for HOPS. The preferred model remains consistent across all five seeds for 3232 and 3939 classes under WRSS, versus 22 and 1717 under HOPS.

We hope the revised manuscript makes both the narrower methodological contribution and its practical diagnostic value clearer.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary for Area-Chair

The reviews converge on whether WRSS defines a distinct evaluation object or repackages existing scores. The new controls support a narrower claim: WRSS is a bounded, budget-aware diagnostic of cumulative target-conditioned severity displacement.

PTQZ: novelty and practical value. For complete equal-mass profiles on a uniform rank line, WRSS has an exact normalized linear form. We will state this directly. The contribution is the transport-derived severity profile, feasible best/worst endpoints, equal-tier handling, and mass-preserving truncation, not nonlinearity. Added linear, logarithmic, and geometric controls instead apply pointwise positional weights and normalize between ideal and reversed-prefix costs; they neither compute cumulative transport nor preserve omitted mass in WRSS's tail bin.

The trained-model analysis gives a concrete use. On CIFAR-100, global hierarchy-aware scores broadly agree, as they should. Under class-conditioned truncation, published HOPS ties cross-entropy and HierCOS for 90/100 classes at k=5, with median class-seed clipping of 100% for both families. WRSS ties only 3 classes and retains graded variation; its preference is consistent across all five seeds for 32 classes, versus 2 for HOPS. At k=20, WRSS resolves 97 comparisons versus 67, with all-seed consistency for 39 versus 17. Full-reference unclipped HOPS recovers much of this resolution, isolating the published reference and clipping as the cause. WRSS therefore localizes which classes gain or lose ranked hierarchy quality after global averaging hides those effects. Under matched Gaussian pixel noise, WRSS is also less volatile than corrected HOPS from k=20 onward. This supports diagnosis, not a universal model-selection oracle.

8ZRy: normalization and reproducibility. Published HOPS clipping can turn large raw response into apparent zero response. Full-reference HOPS repairs much of this but is not generally bounded for pointwise deviation. Full-ranking OLS is a useful global-order control, while truncation removes its shared-multiset interpretation. The response now defines typical adjacent response, LLIP, oriented AUROC, all perturbation samplers, trained-model pixel noise, and seed aggregation. Range diagnostics are restricted to metrics intended for [0,1].

XcuW: claimed equivalence. HOPS sums pointwise absolute deviations; WRSS sums cumulative prefix imbalances. A three-element counterexample separates their numerators. The proposed linear-HOPS control with full reversal and no clipping also differs on all datasets and under trained-model perturbations. WRSS agrees only with its own transport-derived linear form.

AHD measures average retained severity; HOPS measures structured profile satisfaction; OLS measures global severity-order agreement; WRSS measures cumulative severity displacement. Agreement on broad conclusions is desirable. WRSS adds stable target- and budget-specific resolution where averaging or profile saturation erases it.

The extra page will prioritize evaluation objects, feasible endpoints, mass-preserving truncation, LLIP/AUROC, and sampler semantics, followed by the HOPS/WRSS derivation, one hierarchy figure, and one class-conditioned trained-model table. Full controls, grids, pseudocode, and defaults remain in the appendix. These additions answer PTQZ's and 8ZRy's substantive concerns while directly refuting XcuW's equivalence claim.

We have uploaded a revised manuscript incorporating the clarifications, definitions, and additional analyses discussed in our responses. We kindly invite the reviewers to consult the revision alongside the rebuttals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.