acceptodds
Under review as a conference paper at ICLR 2027

Recovering Rank Axes in Embedding Spaces via Differentiable Ranking

13 views

Abstract

Recent studies have shown that visual embeddings contain latent directions along which projections preserve ordinal attributes such as age, crowd count, and aesthetic quality. These *rank axes* provide an interpretable way to uncover ordinal structure in pretrained representations, yet existing estimation methods do not directly target ordinal agreement. In this paper, we study rank axis estimation in a fixed embedding space through differentiable ranking and ask when minimizing a rank-based objective recovers the underlying ordinal direction. We consider two estimators based on quadratically regularized ranking over the permutahedron, using a Spearman-type squared rank loss and a Fenchel–Young loss. We establish consistency of the Spearman-based estimator under Gaussian noise and of the Fenchel–Young estimator in the noise-free setting. Notably, their behavior under noise differs, showing that closely related differentiable ranking objectives can have different recovery properties. We also establish convergence to first-order stationarity for both optimization procedures. Experiments on synthetic and real-world image datasets demonstrate accurate rank-axis recovery and improved overall and top- ranking performance over linear regression and learning-to-rank baselines. Our results show that differentiable ranking provides an effective framework for recovering ordinal structure in pretrained embeddings, while highlighting the statistical consequences of the choice of ranking objective.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
rankability, ordinal structure, embedding geometry, structured representations

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_EbdaOpenReview ↗
Rating
1
Confidence
3(−1)
Informativeness
2(+1)
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the problem of recovering “rank axes” in vision embedding spaces; rank axes are directions in the embedding space along which images can be arranged by known semantic attributes. While existing work recovers these axes using the linear regression loss w.r.t. ground truth values of the semantic attribute,  this work studies the scenario where “soft ranks” are used as the ground truth, enabling one to employ differentiable objectives directly on the ranking information. Two of these objectives, a Spearman-type squared-rank loss and a Fenchel-Young loss are studied theoretically (w.r.t. their convergence and consistency) and empirically (w.r.t. their empirical performance against linear regression). I will refer to these two objectives as SRank-Sp and SRank-FY from here on out. With experiments using synthetic and real-world datasets, the paper aims to demonstrate that the proposed objectives (particularly, SRank-Sp) recover the underlying rank axes better than existing baselines.

Critical strengths

The paper compares different methods to recover rank axes from vision embeddings; especially with the added baselines suggested by me and the other reviewers, it might potentially be useful for future research.

Critical weaknesses

W1. Insufficient motivation for explicit ranking objectives

  • [Critical] The need for an explicit ranking objective is not sufficiently justified in lines 047-051. Granted that linear regression uses raw attribute values, isn't it a necessarily stricter objective that includes ranking (i.e., it’s not an imprecise proxy optimizing for something entirely different)? Recovering the numerical magnitudes correctly will necessarily recover the correct ranks. Therefore, I’m not convinced that including numerical magnitudes is a significant shortcoming of the baseline, thereby providing the core motivation for the paper.

  • [Critical] Continuing on the point above, one could also consider linear regression over the hard ranks themselves. The proposed methods add complexity in terms of implementation, and an extra hyperparamater lnl_n. I therefore find it reasonable to expect that this additional complexity should be justified in terms of demonstrated superiority over a minimally adjusted baseline (in fact, the original rankability paper does use ranks for datasets where numerical magnitudes are not available). This stands beside my original point: the paper does not clearly establish why using numerical magnitudes is a shortcoming.

  • §4.1: This section is a good sanity check showing that the theoretical consistency and convergence proofs indeed carry over empirically. But the main observation here, i.e., the proposed ranking objectives converge and increasingly recover the original rank axis with more synthetic data, also follows rather directly from §3. It's nevertheless good to see that this happens under finite data, but that is also demonstrated by the real-data experiment in §4.3. Unless there is a deeper insight that I missed, I disagree with including it among the main contributions of the paper (line 023).

W2 (more critical than W1). Advantage of the proposed method over the linear regression baseline was not sufficiently demonstrated.

  • [Critical] §4.3: For the datasets where SRank has large advantages over the linear regression baseline, the reported numbers for the baseline are substantially smaller than those reported in the original rankability work (Sonthalia et al., 2025). Table 1 shows that SRank-Sp substantially outperforms linear regression on the three crowd counting datasets. Statistical strength notwithstanding, Sonthalia et al. report SRCCs of 0.854, 0.749, 0.887 on the three datasets respectively (see Table 2 in their paper), much closer to SRank’s reported numbers than they appear in Table 1 of this submission. Plausibly, this was caused by differences in the data split or hyperparameters, but I did not find discussion about it in the paper.

  • §4.3: error bars over multiple trials are missing from the results; as some differences are quite small (examples: UTKFace, Adience and KonIQ-10k in Table 1), it’s important to check whether the proposed method is still superior to linear regression after accounting for randomness.

  • §4.1: While my concerns regarding the significance of this section remain, the linear regression (OLS) baseline is quite critically needed here and is currently missing. This is crucial for contextualizing the convergence of the 5 methods shown here, 4 of which were introduced in the paper itself.

  • In §4.1, SRank-FY is claimed as the strongest method overall (line 376); §4.3 then shows that SRank-Sp usually outperforms SRank-FY. This somewhat limits the practical takeaways one may gain from the paper: i.e., which of the two should one use in practice? I find it limiting that the paper does not take a stand here.


Given these experimental flaws (W2), the current manuscript does not sufficiently validate its claims. Separately from this, I am not yet convinced of the significance of the research question as a standalone study (W1). I think addressing these concerns would require significant changes to the manuscript. Therefore, I recommend a clear rejection at this stage.

Questions for authors

Please treat my points under Critical Weaknesses as questions, particularly the first point under W2. In addition to these, I will consider the following towards a potential revision of my score.

Q1. The work builds substantially on top of the original rankability paper, Sonthalia et al. However, it does not evaluate some of the datasets used in that work ("Kinect", "HCI", and "AVA"). This brings the number of attributes considered in the paper, down to 3 from 7. Could the authors clarify why the above datasets were omitted (for instance, is the proposed method not compatible with them)?

Q2. Please specify the dataset splits used in the study, and the model selection procedure (hyperparameter search) conducted for the linear regression baseline. This is important for understanding the discrepancies between the original baseline numbers for rankability and those reported in the submission.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Ebda (Part 1)

Thank you for your careful review. Please see our responses to the concerns below.

W1-1

We agree that exact recovery of attribute values necessarily preserves their ordering. Our concern is not that linear regression ignores ranking, but that minimizing squared prediction error need not yield the direction that best preserves the ordering when the attribute values cannot be represented exactly by a linear projection.

In particular, rankability depends only on the ordering of attribute values and is invariant to strictly increasing transformations of those values. Such transformations preserve the set of order-preserving axes but can change the least squares regression direction. Thus, regression imposes additional requirements on the numerical magnitudes and spacing that are not necessary for rank axis recovery.

Our motivation is therefore to align the estimation objective directly with the definition of rankability, rather than to suggest that numerical supervision is inherently disadvantageous. We will revise the Introduction to clarify this distinction.

W1-2

We agree that linear regression on hard ranks is a natural and important baseline, as it removes dependence on the original attribute magnitudes without introducing a differentiable ranking operator.

The key distinction is that rank regression minimizes the squared difference between projected scores and numerical rank targets, whereas SRank-Sp compares the soft ranks of the projected scores with the target ranks. Rankability requires agreement in ordering, not agreement in numerical spacing. Under quadratic regularization, soft ranks coincide with hard ranks when all adjacent projected score gaps are at least lnl_n (Blondel et al., 2020, Appendix B.3). Consequently, correctly ordered and sufficiently separated scores can achieve zero SRank-Sp loss without matching the numerical rank values. The same condition also yields zero SRank-FY loss.

To determine whether this distinction leads to a practical advantage, we have added linear regression on hard rank targets as an additional baseline, following the same experimental protocol. The results show that the proposed ranking approach retains its advantage over this baseline, indicating that applying a rank transformation to the predicted scores provides benefits beyond transforming the targets alone. We will include hard rank regression and the other additional baselines in the revised manuscript. Please see the table in our response to W2-1&Q2 for the full comparison.

W1-3

We agree that the synthetic experiments primarily serve to validate the theoretical findings under controlled finite-sample settings, rather than constitute an independent methodological contribution. Their main purpose is to evaluate recovery of a known ground-truth axis under varying sample sizes, dimensions, and noise levels, which cannot be directly assessed on real-world datasets. We will therefore revise the contribution statement to avoid presenting the expected synthetic recovery behavior as a separate novel finding. Instead, we emphasize the statistical recovery analysis and the empirical improvements observed on real-world ranking tasks, while retaining the synthetic experiments as supporting evidence.

AuthorsOpenReview ↗

Response to Reviewer Ebda (Part 2)

W2-1&Q2

  • Comparison with published results (Sonthalia et al., 2025): The cited values 0.854, 0.749, and 0.887 correspond to nonlinear MLP. The corresponding linear regression results are 0.843, 0.734, and 0.869.

  • Embeddings and dataset splits: We extracted the embeddings using the code released by Sonthalia et al. (2025). We obtained the datasets and split information from the sources specified in Table 1 of Sonthalia et al. (2025). For UTKFace, we use the train and test partition from the prior work cited there. For Adience, we follow the official fold assignments, while UCF-QNRF, ShanghaiTech-A/B, and KonIQ-10k use their official dataset partitions.

DatasetTrainValidationTest
UTKFace11,8331,3143,287
Adience13,8471,4323,343
UCF-QNRF1,081120334
ST-A27030182
ST-B36040316
KonIQ-10k7,0581,0002,015
  • Regression and model selection: Linear regression is trained on frozen embeddings using MSE and Adam with weight decay. We follow their hyperparameter search ranges and test 30 random configurations for each dataset and model. The learning rate is sampled log-uniformly from [10−6,10−1][10^{-6}, 10^{-1}]. Weight decay is sampled log-uniformly from [10−7,10−4][10^{-7}, 10^{-4}]. Horizontal flipping is randomly enabled or disabled. Each run uses 100 epochs and a batch size of 128 with cosine learning rate decay. The checkpoint within each run is selected by the highest validation SRCC. The regression baseline in the submission selected the final configuration among the 30 trials using validation MAE. In the revised experiments, we change this final selection criterion to validation SRCC. This improves the regression baseline primarily on UCF-QNRF and ST-A.

  • Additional baselines: We additionally evaluate regression using log- and rank-transformed attributes (Log Ridge, Rank Ridge in the table, respectively) to assess sensitivity to the numerical scale of the attributes. We also include three additional methods suggested by other reviewers, namely RankNet, PiRank, and SmoothI. Both checkpoint selection within each run and final configuration selection for this baseline use validation SRCC. The table reports mean test SRCC across the same seven encoders for all methods. Raw (MAE) and Raw (SRCC) denote regression on the original attributes with final configurations selected by validation MAE and validation SRCC, respectively. For SRank-Sp and SRank-FY, we report mean (SD) across 20 random initializations, averaging SRCC over the seven embedding models in each run. (Given the limited time available to address comments from five reviewers, we were unable to conduct repeated experiments for the other methods in this particular comparison. We will include results from repeated runs in the revised manuscript.)

DatasetRaw (MAE)Raw (SRCC)Log RidgeRank RidgeRankNetListNetPiRankSmoothISRank-SpSRank-FY
UTKFace0.76380.76300.74930.67360.66980.69080.64070.71690.7773(0.0005)0.7019(0.0100)
Adience0.90910.90910.91100.86640.87900.88130.46200.87900.9079(0.0007)0.9030(0.0009)
UCF-QNRF0.65750.69670.81040.76190.69170.29310.61850.69990.8603(0.0042)0.8693(0.0029)
ST-A0.43690.50840.66970.72000.64770.48780.61500.65190.7541(0.0171)0.7436(0.0159)
ST-B0.80250.80230.85530.77320.72040.61290.75440.78770.8829(0.0057)0.8873(0.0055)
KonIQ-10k0.78280.78320.72230.57150.65820.67930.75580.71140.7988(0.0004)0.7937(0.0001)

Log-transforming the targets brings our regression results closer to the published values, although the remaining discrepancy has not been fully explained. SRank-Sp achieves higher mean SRCC than all regression baselines on every data except Adience. We will add the dataset split sizes to Table 3 and the regression training and selection details to Appendix A.1.2, and update the results to include SRCC-based model selection and log- and rank-target regression baselines.

AuthorsOpenReview ↗

Response to Reviewer Ebda (Part 3)

W2-2

We evaluated SRank-Sp and SRank-FY over 20 random initializations while keeping the data split fixed. The two tables below report test SRCC as mean (SD) for every dataset and embedding model.

[SRank-Sp]

Dataset/EmbeddingRN50ViTB32CNXDINO-B14CLIP-RN50CLIP-ViTB32CLIP-CNX
UTKFace0.6653(0.0028)0.7660(0.0019)0.7913(0.0008)0.7739(0.0007)0.8025(0.0002)0.8145(0.0006)0.8280(0.0004)
Adience0.8299(0.0018)0.8894(0.0003)0.8983(0.0040)0.9163(0.0017)0.9313(0.0001)0.9442(0.0001)0.9460(0.0005)
UCF-QNRF0.8626(0.0086)0.8498(0.0116)0.8585(0.0093)0.8458(0.0098)0.8804(0.0104)0.8643(0.0076)0.8606(0.0108)
ST-A0.7563(0.0296)0.8109(0.0532)0.7809(0.0160)0.7006(0.0211)0.7452(0.0336)0.7410(0.0588)0.7436(0.0408)
ST-B0.9008(0.0067)0.8655(0.0319)0.8789(0.0040)0.8740(0.0027)0.8953(0.0093)0.8748(0.0161)0.8909(0.0065)
KonIQ-10k0.7566(0.0023)0.7499(0.0005)0.7787(0.0019)0.7454(0.0019)0.8475(0.0009)0.8357(0.0007)0.8777(0.0016)

[SRank-FY]

Dataset/EmbeddingRN50ViTB32CNXDINO-B14CLIP-RN50CLIP-ViTB32CLIP-CNX
UTKFace0.5539(0.0199)0.7467(0.0259)0.7518(0.0283)0.6098(0.0390)0.6934(0.0231)0.7888(0.0040)0.7690(0.0211)
Adience0.8228(0.0042)0.8894(0.0001)0.9038(<0.0001)0.9039(<0.0001)0.9177(0.0055)0.9374(<0.0001)0.9457(0.0006)
UCF-QNRF0.8711(0.0047)0.8714(0.0060)0.8625(0.0085)0.8573(0.0054)0.8744(0.0061)0.8792(0.0081)0.8695(0.0025)
ST-A0.7805(0.0370)0.7774(0.0270)0.7905(0.0220)0.7222(0.0281)0.7289(0.0612)0.6918(0.0418)0.7142(0.0906)
ST-B0.9013(0.0144)0.8833(0.0178)0.8758(0.0304)0.8640(0.0128)0.9012(0.0107)0.8934(0.0028)0.8922(0.0090)
KonIQ-10k0.7433(<0.0001)0.7507(0.0002)0.7744(<0.0001)0.7496(0.0003)0.8352(<0.0001)0.8263(0.0001)0.8764(<0.0001)

Compared with the strongest regression baseline in W2-1&Q2, SRank-Sp has higher mean SRCC on UTKFace and KonIQ-10k. SRank-FY is higher only on KonIQ-10k. Neither method is higher on Adience. We will update Table 1 with the averaged results and Tables 4 and 5 with the detailed means and SDs.

Q1

The method can be applied to Kinect, HCI, and AVA. We omitted them from the initial evaluation for practical reasons.

  • Kinect: Sonthalia et al. (2025) randomly held out six sequences for testing. We had not matched those sequence identities for direct comparison with their reported results.
  • HCI: HCI provides five decade categories. We initially focused on datasets with more distinct label values. Our method is applicable to HCI using average rank labels.
  • AVA: AVA was excluded from our experiments because processing approximately 230,000 training and validation images exceeded the memory constraints of our current implementation, precluding a comparison with the other methods.
Reviewer_hiXiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper estimates a rank axis: a unit direction v in a fixed pretrained embedding space such that projections vᵀf preserve the order of a continuous attribute (age, crowd count, image quality). Prior work (Sonthalia et al., 2025) fits this direction with linear regression or extreme-sample differences. This paper instead optimizes ordinal agreement directly, using quadratically regularized soft ranks over the permutahedron (Blondel et al., 2020). It considers two objectives: a Spearman-type squared soft-rank loss and the corresponding Fenchel–Young (FY) loss. Both are optimized by Riemannian gradient descent on the sphere.

Experiments use synthetic Gaussian data, measuring axis recovery against soft top-k variants and ListNet. They also use six image datasets (UTKFace, Adience, UCF-QNRF, ShanghaiTech A/B, KonIQ-10k) with seven encoders. The rank-based estimators improve SRCC and top-k metrics over linear regression and ListNet, most clearly on crowd counting.

Critical strengths
  1. The paper shows that two differentiable ranking losses built on the same soft-rank operator differ statistically. The Spearman loss is consistent under noise. The population FY loss favors directions with low projected dispersion and can be biased under noise (Example 1, App. A.4, Figs. 4–5). This matters because FY losses are often chosen for their smoothness alone, and this is a concrete, well-explained counterexample.

  2. The paper argues that rank axes should be fit by targeting ordinal agreement rather than attribute magnitudes, and the proofs are carefully done. Lemma 1 links the l_n = γ/n scaling to exact hard ranks at population quantiles. The identifiability argument (comonotonicity, Lemmas 2–3) is tidy. On real data, gains over regression on crowd counting are large and consistent across seven encoders (e.g., ST-A 0.77 vs 0.44, UCF-QNRF 0.87 vs 0.66).

Critical weaknesses
  1. The theory does not show any setting where rank-based estimation has an advantage over regression, which is the paper’s central motivation
  2. It is unclear whether the reported improvement comes from the ranking objective or because the weak baselines.
Questions for authors
  1. How do Theorem 3 and its proof relate to maximum rank correlation estimation (Han, 1987; Sherman, 1993; Cavanagh & Sherman, 1998)?
  2. Is the linear regression baseline regularized?
1 comment
AuthorsOpenReview ↗

Response to Reviewer hiXi

Thank you for your careful review. Please see our responses to the concerns below.

W1

As discussed in our response to Reviewer Ebda's W1-1, rankability requires preserving attribute ordering rather than reproducing numerical values. When the linear model is misspecified, minimizing squared prediction error need not yield the direction that best preserves the ordering.

Theorems 3 and 4 establish consistency of SRank-Sp under Gaussian noise and SRank-FY in the noise-free setting, respectively. We acknowledge that OLS is also consistent under the correctly specified linear model and standard identifiability conditions. Thus, these theoretical results do not establish superiority over regression.

To further address this concern, we conducted an additional experiment with a nonlinear monotone transformation of the attribute values, where the linear regression model is misspecified but the underlying ordering is preserved. Please refer to the final two tables in our response to Reviewer Ebda's W2–3 for the results. The proposed methods achieve better finite sample axis recovery and ranking performance than OLS, providing empirical evidence of the advantage of directly optimizing ordinal agreement in this setting.

W2

To address this concern, we have added several baselines, including Log Ridge (regression on log-transformed attributes), Rank Ridge (regression on rank-transformed attributes), RankNet, PiRank, and SmoothI. We have also changed the criterion for selecting the final regression configuration from validation MAE to validation SRCC. The table reports mean test SRCC across seven embedding models. Raw (MAE) and Raw (SRCC) denote the final configuration selection criterion.

DatasetRaw (MAE)Raw (SRCC)Log RidgeRank RidgeRankNetListNetPiRankSmoothISRank-SpSRank-FY
UTKFace0.76380.76300.74930.67360.66980.69080.64070.71690.77730.7019
Adience0.90910.90910.91100.86640.87900.88130.46200.87900.90790.9030
UCF-QNRF0.65750.69670.81040.76190.69170.29310.61850.69990.86030.8693
ST-A0.43690.50840.66970.72000.64770.48780.61500.65190.75410.7436
ST-B0.80250.80230.85530.77320.72040.61290.75440.78770.88290.8873
KonIQ-10k0.78280.78320.72230.57150.65820.67930.75580.71140.79880.7937

Log-transforming the targets improves regression on all three crowd counting datasets. Both SRank-Sp and SRank-FY retain higher mean SRCC across the seven encoders on these datasets. We will include these updated comparisons in the revised manuscript.

Q1

We thank the reviewer for highlighting this connection. Han (1987), Sherman (1993), and Cavanagh and Sherman (1998) established the classical foundation for rank-based estimation. In particular, with rank weights and no ties, the Spearman criterion of Cavanagh and Sherman (1998) is equivalent to our hard squared-rank loss, up to an additive and multiplicative constant. We will cite these works and clarify that we do not claim a new rank-correlation principle.

Theorem 3 establishes almost-sure consistency of global minimizers of the quadratically regularized soft-rank loss under Assumptions (A1)–(A6), with ln=γ/nl_n=\gamma/n. The additional analysis controls the sample-size-dependent soft-rank approximation uniformly over candidate directions (Lemmas 1 and 4–5), which does not follow directly from convergence of soft ranks to hard ranks for a fixed score vector. We do not claim greater generality or new asymptotic distributional guarantees beyond the classical literature.

We agree (with Reviewer gYJY) that the noise-free identification in Theorem 4 (Lemma 9) is relatively straightforward. Its role is to provide a reference case for the comparison under noise. The key distinction is that SRank-Sp remains consistent under the stated Gaussian noise assumptions, whereas Example 1 demonstrates that the Fenchel--Young population objective can favor an incorrect direction. Thus, objectives based on the same soft-ranking operator need not have the same population minimizer or recovery properties.

We will revise the related-work discussion and contribution statements accordingly.

Q2

Yes. The linear regression baseline uses weight decay, with its coefficient sampled log-uniformly from [10−7,10−4][10^{-7}, 10^{-4}] as part of the 30 configuration hyperparameter search. Please see our response to Reviewer Ebda, W2-1&Q2 for the full training and model selection details.

Reviewer_gYJYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper estimates a rank axis, i.e. a unit direction in a frozen embedding space whose projections preserve the order of an attribute (age, crowd count, image quality), by directly minimizing soft-rank losses over the permutahedron: a squared soft-rank ("Spearman") loss and a Fenchel–Young (FY) loss, optimized by projected gradient descent on the sphere. Theoretically, it shows convergence to stationary points for both losses, consistency of the global minimizer under the linear model a=⟨v∗,f⟩+εa=\langle v^*,f\rangle+\varepsilon (Spearman with Gaussian noise, FY only without noise), and an example where FY fails under noise. Empirically, it evaluates axis recovery on synthetic data and ranking quality on six image datasets × seven encoders, outperforming linear regression and ListNet on SRCC and top-kk metrics on most datasets.

Critical strengths
  1. Well-posed question, rigorous answer. Asking when a differentiable ranking loss actually recovers the rank axis is a good question, and the Spearman consistency proof is correct as far as I checked (Lemmas 1–6).

  2. Informative negative result. Two losses built on the same soft-rank operator have different population minimizers under noise. This is a useful warning for practitioners who treat them as interchangeable.

  3. Simple and broadly evaluated. The method is a cheap plug-in on frozen embeddings, tested on 6 datasets × 7 encoders with both global and top-kk metrics.

Critical weaknesses

W1. The theory gives no reason to prefer ranking over regression. The paper motivates ranking losses by arguing that regression depends on the scale and spacing of the labels. Yet the consistency results assume (A6), a linear model with Gaussian noise, under which OLS (the criticized baseline) is itself consistent. OLS/ridge may matched or beat SRank curve, but OLS is omitted there. The case where ranking should help (labels that are an unknown monotone transform of the projection, with non-Gaussian embeddings) is not analyzed or tested.

W2. The main consistency result is close to classical work. Estimating a linear direction by maximizing rank correlation is well studied in econometrics (Han, 1987; Sherman, 1993; Cavanagh & Sherman, 1998). With rank weights, the Cavanagh–Sherman estimator is exactly the non-smoothed Spearman loss, and its consistency is proved for more general monotone index models; none of this work is cited. Moreover, the choice ln=γ/nl_n=\gamma/n makes the soft rank equal the hard rank in the limit (Lemma 1), so the new part is mainly bounding the soft-vs-hard approximation error. The identification step behind Theorem 4 (FY, noise-free) is nearly immediate (Lemma 9).

W3. The theorems are about global minimizers, but the algorithm does not find them. Theorems 3–4 concern the exact minimizer of a nonconvex loss, whereas Theorems 1–2 only guarantee stationary points. The 2-D check in App. A.4 exposes this gap: for σ≥1\sigma\ge 1 and n≤500n\le 500, the returned direction stays at cosine ≈0.707\approx 0.707 with the true axis (45° off, likely the initialization), and Fig. 5b shows that the true axis has a lower loss than the returned solution. So these figures do not validate the theory.

W4. The linear-regression baseline looks much weaker than in prior work. With the same encoders, datasets and embedding code, Sonthalia et al. (2025) report far higher regression SRCC on crowd counting (ST-A 0.734 vs. 0.437 here; UCF-QNRF 0.843 vs. 0.658). Against the published numbers, SRank-Sp gains only about 0.02–0.03, not the 0.08–0.33 reported. A likely cause is regression on raw, heavy-tailed counts without a log transform or proper regularization. The 16 s training time for a ~300×768 regression (milliseconds in closed form) also suggests an unusual implementation.

W5. Key baselines are missing. Pairwise linear ranking, the classical way to learn a ranking direction (RankSVM for relative attributes, Parikh & Grauman, 2011; or RankNet), is not compared. Neither is ridge regression on rank- or log-transformed labels, a simple fix for the paper's own criticism of regression.

W6. The Spearman-vs-FY conclusion is not backed by experiments. The paper says FY's failure under noise "motivates" the Spearman loss. But by my calculation (Q4, see below), for Gaussian embeddings FY is biased only when their covariance is anisotropic. On the isotropic Gaussian data of Sec. 4.1 it stays consistent, which explains why SRank-FY is actually the best method in Figs. 1–2. On real data the two losses perform similarly on average, and no experiment shows FY failing as predicted.

Overall, W1–W3 weaken the main (theoretical) contribution, and W4–W6 mean the empirical improvements are not reliably established.

Questions for authors

Q1. Regression baseline. How was regression implemented (closed form or SGD, regularization, target transform)? Why is it much weaker than in Sonthalia et al. (e.g., RN50 on ST-A: 0.085 vs. 0.799), and why does it take 16 s? Please add ridge regression on log/rank targets; if SRank-Sp still clearly wins, I will raise my score.

Q2. Baselines. Please report the dataset split sizes, how the ListNet targets were built, and a linear RankSVM/RankNet baseline.

Q3. App. A.4. Which initialization and step sizes were used? Is the 0.707 plateau the initialization? Please report the exact 2-D grid-search minimizer instead.

Q4. FY bias. For Gaussian design f∼N(0,Σ)f\sim N(0,\Sigma), I find that the population FY objective is ∝v⊤Σv (1−ρσ corr(v⊤f,v∗⊤f))\propto\sqrt{v^\top\Sigma v}\,\bigl(1-\rho_\sigma\,\mathrm{corr}(v^\top f, v^{*\top}f)\bigr), with ρσ=τ∗/τ∗2+σ2\rho_\sigma=\tau_*/\sqrt{\tau_*^2+\sigma^2} and τ∗2=v∗⊤Σv∗\tau_*^2=v^{*\top}\Sigma v^*. So FY is consistent if Σ∝I\Sigma\propto I, but otherwise biased toward low-variance directions. Can you confirm this, and check whether whitening removes the large FY/Sp gaps on ST-A (CLIP-ViTB32/CLIP-CNX)?

Q5. Regularization. The theory sets ln=γ/nl_n=\gamma/n with γ=1/K\gamma=1/K (KK an unknown density bound), but the experiments tune ll on a grid independent of nn. Are the selected values covered by the theory?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gYJY (Part 1)

Thank you for your careful review. Please see our responses to the concerns below.

W1

We agree that OLS is also consistent under Assumption (A6) and standard identifiability conditions, and that our consistency results do not establish superiority over regression. As discussed in our response to Reviewer Ebda's W1-1, minimizing squared prediction error need not yield the direction that best preserves attribute ordering when the linear model is misspecified.

We have also included OLS in the original synthetic experiments. As expected, OLS performs strongly under the correctly specified linear model. To address the reviewer's suggestion, we conducted an additional experiment using non-Gaussian embeddings sampled uniformly from a ball and attributes generated through a nonlinear monotone transformation. Both SRank-Sp and SRank-FY achieve higher cosine similarity and test SRCC than OLS across the evaluated settings. These results demonstrate finite sample advantages of directly optimizing ordinal agreement under model misspecification, without establishing general asymptotic superiority over regression. The results for both the original synthetic setting and the additional nonlinear setting are provided in our response to Reviewer Ebda's W2-3.

W2

Please see our response to Reviewer hiXi’s Q1 for how our consistency theorem and its proof relate to these classical rank estimators.

Q1&W4&W5 (Rank- or Log-transformed labels)

We added regression and ranking baselines, including Log Ridge (regression on log-transformed attributes), Rank Ridge (regression on rank-transformed attributes), RankNet, PiRank, and SmoothI, to reassess the reported gains. We also evaluated SRank-Sp and SRank-FY over 20 random initializations and use their mean results in the updated comparison. (Given the limited time available to address comments from five reviewers, we were unable to conduct repeated experiments for the other methods in this particular comparison. We will include results from repeated runs in the revised manuscript.)

Our regression baseline follows the official implementation of Sonthalia et al. (2025). It fits a linear model using MSE and minibatch Adam with weight decay. For each dataset and embedding pair, we test 30 configurations with learning rate in [10−6,10−1][10^{-6},10^{-1}] and weight decay in [10−7,10−4][10^{-7},10^{-4}], both log-uniformly, and horizontal flipping randomly enabled or disabled. Each run uses 100 epochs, batch size 128, and cosine learning rate decay. We did not use a closed-form solver. The reported 16 s is the runtime of one Adam training run.

The submission used validation MAE to select the final regression configuration. We now use validation SRCC. This improves regression performance on UCF-QNRF and ST-A. For the RN50 on ST-A case, test SRCC increases from 0.0849 with MAE-based configuration selection to 0.4168 with SRCC-based selection. Log Ridge and Rank Ridge achieve 0.7413 and 0.7604, closer to the published value of 0.799.

The table reports mean test SRCC across seven embedding models. Raw (MAE) and Raw (SRCC) distinguish the final selection criteria for the original targets. Log Ridge and Rank Ridge use transformed targets and SRCC-based selection. Detailed means and SDs for SRank on each dataset and embedding model are provided in our response to Reviewer QZ1E, Q3.

DatasetRaw (MAE)Raw (SRCC)Log RidgeRank RidgeRankNetListNetPiRankSmoothISRank-SpSRank-FY
UTKFace0.76380.76300.74930.67360.66980.69080.64070.71690.77730.7019
Adience0.90910.90910.91100.86640.87900.88130.46200.87900.90790.9030
UCF-QNRF0.65750.69670.81040.76190.69170.29310.61850.69990.86030.8693
ST-A0.43690.50840.66970.72000.64770.48780.61500.65190.75410.7436
ST-B0.80250.80230.85530.77320.72040.61290.75440.78770.88290.8873
KonIQ-10k0.78280.78320.72230.57150.65820.67930.75580.71140.79880.7937

Log Ridge brings our results on UCF-QNRF, ST-A, and ST-B closer to the published values. We used the released code in Sonthalia et al. (2025) but did not match the exact checkpoints and random seeds of the published experiments. These differences may partly explain the remaining gap, but their effect has not been confirmed. SRank-Sp achieves higher mean SRCC than Rank Ridge on all six datasets and Log Ridge on five datasets. We will include the updated comparisons in the revised manuscript and document the implementation in Appendix A.1.2.

AuthorsOpenReview ↗

Response to Reviewer gYJY (Part 2)

Q2&W5 (Pairwise linear ranking)

All methods and embedding models are evaluated using the same training, validation, and test sets. We follow the split sources referenced in Table 1 of Sonthalia et al. (2025). The table below provides the sample counts for each split. We will also add these split sizes to Table 3 of the revised manuscript.

DatasetTrainValidationTest
UTKFace11,8331,3143,287
Adience13,8471,4323,343
UCF-QNRF1,081120334
ST-A27030182
ST-B36040316
KonIQ-10k7,0581,0002,015

For ListNet, attribute labels were centered and scaled using their respective training set means and average centered l2l_2 norms as in SRank. We applied softmax to the normalized labels to construct the target distributions and minimized cross-entropy between these targets and the softmax of the linear prediction scores.

For the pairwise ranking comparison requested in W5, we included a linear model trained with RankNet loss. To manage the computational cost of the large number of image pairs, we optimized the pairwise logistic loss using minibatch SGD and sampled at most 250,000 candidate pairs per dataset. RankSVM was omitted due to its long training time. Please see the table in our response to Q1 for the results of RankNet, other additional baselines, SRank-Sp and SRank-FY.

Q3&W3&W6

The original experiments initialized the direction at (1,1)/2(1,1)/\sqrt{2} and considered initial step sizes 0.5,0.1,0.05,0.01,0.005,0.001\\{0.5,0.1,0.05,0.01,0.005,0.001\\}. We confirmed that the approximately 0.707 plateau at small sample sizes corresponds to the initial direction. We agree that these optimization outputs should not be interpreted as global minimizers.

Following the reviewer's suggestion, we evaluated both empirical objectives over 3,600 equally spaced directions on the unit circle and selected the minimum loss direction for each objective. This search does not depend on initialization or step sizes. The table below summarizes the results at n=50,000n=50,000. Cosine similarity measures alignment with the true axis, and gap denotes the empirical loss at the true axis minus the minimum grid loss.

σ\sigmaSp cosineSp gapFY cosineFY gap
01.000001.00005.2664×10−65.2664\times10^{-6}
10.99982.8312×10−62.8312\times10^{-6}0.42583.5918×10−23.5918\times10^{-2}
20.99947.3314×10−67.3314\times10^{-6}0.21479.3179×10−29.3179\times10^{-2}

The Spearman grid minimizers closely align with the true axis and have small loss gaps. Under noise, the Fenchel--Young minimizers remain substantially misaligned and achieve lower empirical loss than the true axis. This agrees with our Example 1.

We will replace Figures 4 and 5 with the full grid-search curves and update Appendix A.4. We will clarify that the reported solutions minimize the loss over the specified finite grid.

Q4&W6 (Gaussian design and whitening)

As noted in the review, the true axis minimizes the population FY objective in the isotropic Gaussian setting. Anisotropy can favor directions with lower projected variance. Whitening with the true population covariance removes this effect.

ST-A has 270 training data and the embedding dimensions of CLIP-ViTB32 and CLIP-CNX are 512 and 768, so the empirical covariance is singular. We therefore use shrinkage whitening Σ^α=(1−α)Σ^+αtr⁡(Σ^)I/d\hat{\Sigma}_{\alpha}=(1-\alpha)\hat{\Sigma}+\alpha\operatorname{tr}(\hat{\Sigma})I/d with α=0.5\alpha=0.5 and apply Σ^α−1/2\hat{\Sigma}_{\alpha}^{-1/2} to the embeddings. The transformation is estimated from the training embeddings and applied to all splits. These comparisons use one initialization seed.

For CLIP-ViTB32, Sp/FY test SRCC changes from 0.7902/0.5886 to 0.7327/0.7615. For CLIP-CNX, it changes from 0.7885/0.5119 to 0.6722/0.7924. FY improves and Sp declines in both cases. The absolute gaps decrease to 0.0288 and 0.1202, respectively. The Sp/FY gaps may remain because the real embeddings are not generated under the Gaussian model and shrinkage whitening does not guarantee exact isotropy.

AuthorsOpenReview ↗

Response to Reviewer gYJY (Part 3)

Q5

The original Gaussian experiment does not satisfy compact support assumption of the consistency results. For real data, the embedding distribution and density bound KK are unknown, so selected regularization values are not guaranteed to satisfy ln=γ/nl_n=\gamma/n.

We therefore added experiments using embeddings fi=d+2uif_i=\sqrt{d+2}u_i with ui∼Unif⁡(Bd(0,1))u_i\sim\operatorname{Unif}(B_d(0,1)) and attributes ai=⟨v∗,fi⟩+εia_i=\langle v^\ast,f_i\rangle+\varepsilon_i for a randomly sampled unit direction v∗v^\ast and independent εi∼N(0,σ2)\varepsilon_i\sim N(0,\sigma^2). This embedding distribution has compact support, covariance IdI_d, and projected densities bounded by Kd=Γ(d/2+1)π(d+2)Γ((d+1)/2)K_d=\frac{\Gamma(d/2+1)}{\sqrt{\pi(d+2)}\Gamma((d+1)/2)}. The density bound need not be tight. Any fixed 0<γ≤1/Kd0<\gamma\le1/K_d corresponds to a valid bound K=1/γ≥KdK=1/\gamma\ge K_d, so the theory also permits these smaller γ/n\gamma/n schedules. We evaluate d∈32,64,128d\in\\{32,64,128\\} and n∈50,100,200,500,1000n\in\\{50,100,200,500,1000\\} under σ∈0,0.5\sigma\in\\{0,0.5\\}, with 100 repetitions per condition. The results below are for d=64d=64.

The table compares the theoretical values with the most frequently selected ll without noise. The last column reports the percentage of runs selecting l≤ln,theoryl\le l_{n,\text{theory}} for Sp and FY, respectively.

nln,theoryl_{n,\text{theory}}Sp: most selected ll (count)FY: most selected ll (count)Sp/FY (%)
500.0507110.1 (99)0.1 (94)0/6
1000.0253560.05 (95)0.02 (100)5/100
2000.0126780.01 (99)0.005 (68)99/100
5000.0050710.005 (91)0.001 (100)100/100
10000.0025360.001 (100)0.001 (100)100/100
At n=500n=500 and n=1,000n=1,000, all selected values are below the reference. This agreement is observed even though validation tuning does not use the density bound.

The following tables compare axis recovery using validation tuning and the fixed theoretical schedule, reporting cosine similarity with the true axis as mean(SD) for σ∈0,0.5\sigma\in\\{0,0.5\\}. Tuned uses validation selected regularization and theoretical uses the fixed regularization ln,theoryl_{n,\text{theory}}.

[Noise-free (σ=0\sigma=0)]

Method / n501002005001000
SRank-Sp (tuned)0.7875 (0.0511)0.9254 (0.0209)0.9962 (0.0024)0.9995 (0.0003)0.9998 (0.0001)
SRank-Sp (theoretical)0.5228 (0.1122)0.9197 (0.0460)0.9977 (0.0012)0.9994 (0.0005)0.9998 (0.0002)
SRank-FY (tuned)0.8126 (0.0475)0.9978 (0.0013)1.0000 (<0.0001)1.0000 (<0.0001)1.0000 (<0.0001)
SRank-FY (theoretical)0.7169 (0.0641)0.9968 (0.0028)0.9997 (0.0002)1.0000 (<0.0001)1.0000 (<0.0001)

[Noisy (σ=0.5\sigma=0.5)]

Method / n501002005001000
SRank-Sp (tuned)0.6801 (0.0625)0.8445 (0.0353)0.9381 (0.0135)0.9773 (0.0041)0.9881 (0.0021)
SRank-Sp (theoretical)0.4657 (0.1063)0.8032 (0.0484)0.9280 (0.0164)0.9763 (0.0046)0.9886 (0.0022)
SRank-FY (tuned)0.6995 (0.0599)0.8554 (0.0290)0.9424 (0.0131)0.9799 (0.0037)0.9902 (0.0019)
SRank-FY (theoretical)0.6181 (0.0593)0.8183 (0.0430)0.9342 (0.0144)0.9793 (0.0041)0.9901 (0.0019)

The theoretical and tuned settings give similar recovery at larger sample sizes under both noise conditions. The recovery trends are consistent with Theorems 3 and 4 without noise and Theorem 3 for Sp under noise. The noisy FY results remain outside the scope of Theorem 4. We will include these results and verification of Assumptions (A1)--(A6) in the revised manuscript.

Reviewer_QZ1EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to recover rank axes that capture the ordinal relationships of continuous attributes in fixed pretrained embedding spaces. Existing approaches typically rely on linear regression or differences between extreme samples to estimate ranking directions, without directly optimizing ordinal agreement. To address this limitation, the authors propose using differentiable ranking to learn rank axes.

Critical strengths
  1. The paper addresses a clearly defined and practically relevant problem.

  2. The proposed method is simple but well aligned with the objective of ranking tasks.

  3. The theoretical analysis is relatively comprehensive and reveals meaningful differences in the statistical properties of different ranking losses.

  4. The experiments cover a variety of attributes, datasets, and pretrained embedding models.

Critical weaknesses
  1. There is a gap between the theoretical assumptions and the synthetic experimental setup, which limits the direct applicability of the consistency results. The consistency results in Theorems 3 and 4 rely on Assumptions (A1)–(A6), among which Assumption (A6) explicitly requires the embedding distribution to have compact support. However, in the synthetic experiments in Section 4.1, the authors generate embeddings using (fi∼N(0,Id))(f_i \sim \mathcal{N}(0,I_d)). Since the Gaussian distribution has unbounded support, this experimental setup does not satisfy the assumption. Consequently, the data-generating process used in the synthetic experiments does not fully align with the conditions required by the consistency theorems. Although the experimental results demonstrate promising recovery performance under Gaussian embeddings, they cannot be regarded as a direct validation of the theoretical guarantees under their stated assumptions.

  2. The methodological novelty is relatively limited, and the distinction from existing differentiable ranking and learning-to-rank methods could be further clarified.

  3. The statistical consistency results concern global minimizers, whereas the proposed optimization algorithm only guarantees convergence to first-order stationary points, leaving a gap between the theoretical guarantees and practical optimization. The consistency results are established for global minimizers of the empirical objectives. However, the convergence analysis of the proposed optimization algorithm only guarantees convergence to first-order stationarity. Since the optimization problem is nonconvex, reaching a stationary point does not necessarily imply finding a global minimizer or recovering the true rank direction. The authors should further discuss how optimization errors affect the statistical recovery guarantees. Alternatively, additional experiments involving multiple random initializations or analyses of local minima would help assess whether the proposed algorithm can reliably find solutions close to the global optimum.

Questions for authors

Please refer to Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer QZ1E (Part 1)

Thank you for your careful review. Please see our responses to the concerns below.

Q1

The Gaussian embeddings in Section 4.1 do not satisfy the compact support requirement in Assumption A6. We therefore added experiments using embeddings fi=d+2uif_i=\sqrt{d+2}u_i with ui∼Unif⁡(Bd(0,1))u_i\sim\operatorname{Unif}(B_d(0,1)) and attributes ai=⟨v∗,fi⟩+εia_i=\langle v^\ast,f_i\rangle+\varepsilon_i for a randomly sampled unit direction v∗v^\ast and independent εi∼N(0,σ2)\varepsilon_i\sim N(0,\sigma^2). This embedding distribution has compact support, covariance IdI_d, and projected densities bounded by Kd=Γ(d/2+1)π(d+2)Γ((d+1)/2)K_d=\frac{\Gamma(d/2+1)}{\sqrt{\pi(d+2)}\Gamma((d+1)/2)}. We evaluate d∈32,64,128d\in\\{32,64,128\\} and n∈50,100,200,500,1000n\in\\{50,100,200,500,1000\\} under σ∈0,0.5\sigma\in\\{0,0.5\\}, with 100 repetitions per condition. The results below are for d=64d=64.

The following tables compare axis recovery using validation tuning and the fixed theoretical schedule, reporting cosine similarity with the true axis as mean(SD) for σ∈0,0.5\sigma\in\\{0,0.5\\}. Tuned uses validation selected regularization and theoretical uses the fixed regularization ln,theory=1/(Kdn)l_{n,\text{theory}}=1/(K_d n).

[Noise-free (σ=0\sigma=0)]

Method / n501002005001000
SRank-Sp (tuned)0.7875 (0.0511)0.9254 (0.0209)0.9962 (0.0024)0.9995 (0.0003)0.9998 (0.0001)
SRank-Sp (theoretical)0.5228 (0.1122)0.9197 (0.0460)0.9977 (0.0012)0.9994 (0.0005)0.9998 (0.0002)
SRank-FY (tuned)0.8126 (0.0475)0.9978 (0.0013)1.0000 (<0.0001)1.0000 (<0.0001)1.0000 (<0.0001)
SRank-FY (theoretical)0.7169 (0.0641)0.9968 (0.0028)0.9997 (0.0002)1.0000 (<0.0001)1.0000 (<0.0001)

[Noisy (σ=0.5\sigma=0.5)]

Method / n501002005001000
SRank-Sp (tuned)0.6801 (0.0625)0.8445 (0.0353)0.9381 (0.0135)0.9773 (0.0041)0.9881 (0.0021)
SRank-Sp (theoretical)0.4657 (0.1063)0.8032 (0.0484)0.9280 (0.0164)0.9763 (0.0046)0.9886 (0.0022)
SRank-FY (tuned)0.6995 (0.0599)0.8554 (0.0290)0.9424 (0.0131)0.9799 (0.0037)0.9902 (0.0019)
SRank-FY (theoretical)0.6181 (0.0593)0.8183 (0.0430)0.9342 (0.0144)0.9793 (0.0041)0.9901 (0.0019)

The theoretical and tuned settings give similar recovery at larger sample sizes under both noise conditions. The recovery trends are consistent with Theorems 3 and 4 without noise and Theorem 3 for Sp under noise. The noisy FY results remain outside the scope of Theorem 4. We will include these results and verification of Assumptions (A1)--(A6) in the revised manuscript.

Q2

We agree that the distinction from existing methods should be stated more explicitly. Rather than proposing new differentiable ranking operators or loss functions, we build on established methods to study the statistical recovery of a global ordinal direction in a fixed embedding space. Unlike general learning-to-rank methods, our goal is to recover an interpretable rank axis rather than learn a general ranking predictor or modify the embeddings. We establish consistency under the stated assumptions and show that the Spearman and Fenchel--Young objectives can exhibit different recovery behavior under noise. In particular, Example 1 demonstrates that an incorrect direction can attain a lower population Fenchel--Young loss than the true axis.

We will revise the Introduction and Related Work to clarify these distinctions and the scope of our theoretical contributions.

AuthorsOpenReview ↗

Response to Reviewer QZ1E (Part 2)

Q3

We agree that our consistency results concern global minimizers and that first-order stationarity does not guarantee finding them. We therefore examined axis recovery and variation in ranking performance across repeated runs with random initializations.

The synthetic experiments include 100 independent runs with randomly sampled data, true axes, and initial directions. Figure 1 in the original manuscript reports the mean recovery performance and ±1\pm 1 standard deviation across these runs. These experiments assess axis recovery across different samples and initializations.

For real data, we repeated the training and hyperparameter search procedure 20 times for SRank-Sp and SRank-FY. Each run used a random initial direction, while the embeddings and dataset splits remained fixed. The tables below summarize test SRCC as mean(SD) for each dataset and embedding model.

[SRank-Sp]

Dataset/EmbeddingRN50ViTB32CNXDINO-B14CLIP-RN50CLIP-ViTB32CLIP-CNX
UTKFace0.6653(0.0028)0.7660(0.0019)0.7913(0.0008)0.7739(0.0007)0.8025(0.0002)0.8145(0.0006)0.8280(0.0004)
Adience0.8299(0.0018)0.8894(0.0003)0.8983(0.0040)0.9163(0.0017)0.9313(0.0001)0.9442(0.0001)0.9460(0.0005)
UCF-QNRF0.8626(0.0086)0.8498(0.0116)0.8585(0.0093)0.8458(0.0098)0.8804(0.0104)0.8643(0.0076)0.8606(0.0108)
ST-A0.7563(0.0296)0.8109(0.0532)0.7809(0.0160)0.7006(0.0211)0.7452(0.0336)0.7410(0.0588)0.7436(0.0408)
ST-B0.9008(0.0067)0.8655(0.0319)0.8789(0.0040)0.8740(0.0027)0.8953(0.0093)0.8748(0.0161)0.8909(0.0065)
KonIQ-10k0.7566(0.0023)0.7499(0.0005)0.7787(0.0019)0.7454(0.0019)0.8475(0.0009)0.8357(0.0007)0.8777(0.0016)

[SRank-FY]

Dataset/EmbeddingRN50ViTB32CNXDINO-B14CLIP-RN50CLIP-ViTB32CLIP-CNX
UTKFace0.5539(0.0199)0.7467(0.0259)0.7518(0.0283)0.6098(0.0390)0.6934(0.0231)0.7888(0.0040)0.7690(0.0211)
Adience0.8228(0.0042)0.8894(0.0001)0.9038(<0.0001)0.9039(<0.0001)0.9177(0.0055)0.9374(<0.0001)0.9457(0.0006)
UCF-QNRF0.8711(0.0047)0.8714(0.0060)0.8625(0.0085)0.8573(0.0054)0.8744(0.0061)0.8792(0.0081)0.8695(0.0025)
ST-A0.7805(0.0370)0.7774(0.0270)0.7905(0.0220)0.7222(0.0281)0.7289(0.0612)0.6918(0.0418)0.7142(0.0906)
ST-B0.9013(0.0144)0.8833(0.0178)0.8758(0.0304)0.8640(0.0128)0.9012(0.0107)0.8934(0.0028)0.8922(0.0090)
KonIQ-10k0.7433(<0.0001)0.7507(0.0002)0.7744(<0.0001)0.7496(0.0003)0.8352(<0.0001)0.8263(0.0001)0.8764(<0.0001)

Ranking performance is generally stable across 20 runs. The variation on ST-A and ST-B may reflect their small training and validation sets. We will update Tables 1, 4, and 5 with the detailed means and SDs.

Reviewer_p7zJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies rank-axis recovery in fixed pretrained embedding spaces using differentiable Spearman and Fenchel–Young objectives. It analyzes their statistical consistency and optimization convergence and evaluates direction recovery and ranking performance on synthetic and real-world datasets.

Critical strengths
  1. The paper addresses an interesting and timely question: when rank-based objectives can recover an underlying ordinal direction in pretrained embeddings.
  2. The theoretical analysis examines how closely related ranking objectives differ in recovery behavior under noise, while clearly separating statistical recovery from optimization guarantees.
  3. Experiments across synthetic data, real-world datasets, and multiple pretrained encoders provide empirical support for the approach.
Critical weaknesses

W1. The methodological contribution is not clearly distinguished. The current presentation does not clearly distinguish the proposed method from existing linear learning-to-rank approaches, making its methodological novelty difficult to assess.

W2. Limited connection between the theoretical analysis and practical use. The theoretical analysis relies on compact support, a uniform upper bound on projected densities, a specific noise model, and a prescribed regularization scale, whereas the experiments select hyperparameters using a validation set. It remains unclear how these assumptions relate to the experimental settings and how the theoretical regularization scale translates into practical parameter choices.

W3. Limited diversity of the synthetic experiments. The synthetic experiments use isotropic Gaussian embeddings and a linear attribute-generation model. This restricted setting leaves the robustness of rank axis recovery under more varied embedding distributions, such as anisotropic or non-Gaussian distributions, insufficiently evaluated.

W4: Comparison and baseline resutls: Please refer to Q1 and Q2

Questions for authors

Q1. Comparability with previously reported regression results. Are the linear regression results directly comparable to those reported by Sonthalia et al. [1], particularly on ShanghaiTech-A (0.4369 vs. 0.734) and UCF-QNRF (0.6575 vs. 0.843)? Please clarify whether the attribute labels, data splits, encoder checkpoints, preprocessing, tuning, and averaging protocols are matched. If the settings are comparable, what accounts for the substantial performance gaps?

Q2. Comparisons with alternative ranking baselines. Could the authors compare against a representative differentiable ranking objective, such as PiRank [2] or SmoothI [3], and a simple baseline that linearly regresses training-label ranks or quantiles, using the same fixed embeddings, linear scoring function, and comparable hyperparameter tuning? These comparisons would help clarify whether the improvements stem from the proposed formulation or from ranking-aware training or label transformation more generally.

These two experimental concerns are central to my current assessment. I am open to raising my score if the authors convincingly address them in the rebuttal.

[1] Sonthalia et al., On the rankability of visual embeddings. NeurIPS 2025. [2] Swezey et al., PiRank: Scalable Learning To Rank via Differentiable Sorting. NeurIPS 2021. [3] Thonet et al., Listwise Learning to Rank Based on Approximate Rank Indicators. AAAI 2022.

4 comments
AuthorsOpenReview ↗

Response to Reviewer p7zJ (Part 1)

Thank you for your careful review. Please see our responses to the concerns below.

W1

We agree that the distinction from existing methods should be stated more explicitly. We build on established differentiable ranking operators and loss constructions to analyze the statistical recovery of an ordinal direction in a fixed embedding space. We establish consistency under stated assumptions and show that the Spearman and Fenchel--Young objectives can exhibit different recovery behavior under noise. Example 1 demonstrates that an incorrect direction can attain lower population Fenchel--Young loss than the true axis.

We will revise the Introduction and Related Work to explain how our rank-axis formulations build on existing differentiable ranking methods and specify the scope of the accompanying theoretical results.

W2

For real datasets, the density bound KK and noise mechanism are unknown. We therefore added experiments using embeddings fi=d+2uif_i=\sqrt{d+2}u_i with ui∼Unif⁡(Bd(0,1))u_i\sim\operatorname{Unif}(B_d(0,1)) and attributes ai=⟨v∗,fi⟩+εia_i=\langle v^\ast,f_i\rangle+\varepsilon_i for a randomly sampled unit direction v∗v^\ast and independent εi∼N(0,σ2)\varepsilon_i\sim N(0,\sigma^2). This embedding distribution has compact support, covariance IdI_d, and projected densities bounded by Kd=Γ(d/2+1)π(d+2)Γ((d+1)/2)K_d=\frac{\Gamma(d/2+1)}{\sqrt{\pi(d+2)}\Gamma((d+1)/2)}.

Please see our response to Reviewer gYJY's Q5 for how the theoretical assumptions and regularization relate to our experimental settings.

W3

We expanded the synthetic evaluation using the non-Gaussian embedding distribution described in our response to W2. We consider ai=(⟨v∗,fi⟩+εi)3a_i=(\langle v^\ast,f_i\rangle+\varepsilon_i)^3. This transformation preserves the ordering of the noisy attributes while changing their numerical scale.

We compare SRank-Sp, SRank-FY, OLS, and Rank OLS on the same data splits. The tables report cosine similarity with the true axis as mean(SD) over 100 repetitions for d=64d=64.

[Noise-free (σ=0\sigma=0)]

Method / n501002005001000
SRank-Sp0.7875 (0.0511)0.9255 (0.0209)0.9962 (0.0023)0.9995 (0.0003)0.9998 (0.0001)
SRank-FY0.8126 (0.0475)0.9978 (0.0013)1.0000 (<0.0001)1.0000 (<0.0001)1.0000 (<0.0001)
OLS0.5179 (0.0984)0.7278 (0.0750)0.8972 (0.0233)0.9618 (0.0101)0.9817 (0.0034)
Rank OLS0.7899 (0.0705)0.9555 (0.0258)0.9897 (0.0039)0.9968 (0.0010)0.9986 (0.0003)

[Noisy (σ=0.5\sigma=0.5)]

Method / n501002005001000
SRank-Sp0.6801 (0.0625)0.8445 (0.0353)0.9382 (0.0134)0.9773 (0.0042)0.9882 (0.0021)
SRank-FY0.6995 (0.0599)0.8554 (0.0290)0.9424 (0.0131)0.9799 (0.0037)0.9902 (0.0019)
OLS0.4123 (0.1023)0.6179 (0.0905)0.8345 (0.0350)0.9362 (0.0134)0.9688 (0.0059)
Rank OLS0.5572 (0.1104)0.7985 (0.0512)0.9358 (0.0145)0.9789 (0.0044)0.9899 (0.0019)

Both SRank methods achieve higher cosine similarity than OLS across the reported conditions. SRank-FY also gives higher cosine similarity than SRank-Sp in these experiments. Rank OLS remains competitive with SRank-Sp and exceeds it in some settings. Similar performance patterns are observed at the other dimensions and in test SRCC. We will include the complete results in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer p7zJ (Part 2)

W4&Q1

We used the official embedding extraction code and followed the dataset sources and regression training procedure of Sonthalia et al. (2025). However, the checkpoints used for their published results were not provided, so we could not verify that our extracted embeddings exactly matched those used in the published experiments.

For UTKFace, we used the partition cited in Table 1 of Sonthalia et al. (2025). Adience follows the official fold assignments. UCF-QNRF, ShanghaiTech-A/B, and KonIQ-10k use their official dataset partitions. The splits are fixed across all methods and encoders in our comparisons.

Regression fits a linear model using MSE and minibatch Adam with weight decay. We tested 30 configurations per dataset and embedding model. Learning rate and weight decay were sampled log-uniformly from [10−6,10−1][10^{-6},10^{-1}] and [10−7,10−4][10^{-7},10^{-4}], respectively. Horizontal flipping was randomly enabled or disabled. Training used 100 epochs, batch size 128, and cosine learning rate decay.

The submission used validation MAE to select the final regression configuration. We now use validation SRCC. This improves regression primarily on UCF-QNRF and ST-A. We also evaluate log- and rank-target regression, which further reduce the gaps with the published results. Differences in checkpoints and random seeds may contribute to the remaining discrepancy.

Q2 below reports the full comparison as mean test SRCC across seven embedding models. We will describe the dataset splits and experiment procedures in Appendix A.1.2, and include the updated comparisons.

W4&Q2

We have added PiRank and SmoothI as differentiable ranking baselines and Rank Ridge (regression on training label ranks), along with Log Ridge (regression on log-transformed attributes) and RankNet, as suggested by other reviewers. All comparisons use the same embeddings and fixed dataset splits. Because our goal is to identify a rank axis, we replace the original nonlinear scorers in PiRank and SmoothI with a linear function si=v⊤fis_i=v^\top f_i. We use the loss implementations provided in the official PiRank and SmoothI repositories.

For PiRank and SmoothI, we construct fixed ranking lists of at most 200 samples within each split. Large raw attribute values can cause numerical overflow in the exponential gain used by these objectives. We therefore map relevance labels to [0,1][0,1] using training set minimum and maximum values. The training budgets are based on published experimental settings. PiRank uses 100,000 Adam updates per candidate, and SmoothI uses 50 epochs. We search seven learning rates for PiRank and eight combinations of learning rate and smoothing parameter for SmoothI. We use validation SRCC for hyperparameter selection to align the baselines with our rank axis evaluation metric. Rank Ridge follows the regression training and search procedure described in Q1, using training label ranks as targets and validation SRCC for selection.

The table reports mean test SRCC across seven embedding models. SRank-Sp and SRank-FY results also average over 20 random initializations. Raw (MAE) and Raw (SRCC) denote regression with final configurations selected by validation MAE and validation SRCC, respectively.

DatasetRaw (MAE)Raw (SRCC)Log RidgeRank RidgeRankNetListNetPiRankSmoothISRank-SpSRank-FY
UTKFace0.76380.76300.74930.67360.66980.69080.64070.71690.77730.7019
Adience0.90910.90910.91100.86640.87900.88130.46200.87900.90790.9030
UCF-QNRF0.65750.69670.81040.76190.69170.29310.61850.69990.86030.8693
ST-A0.43690.50840.66970.72000.64770.48780.61500.65190.75410.7436
ST-B0.80250.80230.85530.77320.72040.61290.75440.78770.88290.8873
KonIQ-10k0.78280.78320.72230.57150.65820.67930.75580.71140.79880.7937

SRank-Sp achieves higher mean SRCC than the regression and ranking baselines on five of the six datasets. We will include these comparisons and the implementation details in the revised manuscript.

Thank the authors for their response. My main concerns have been adequately addressed, and I have raised my score accordingly.

We sincerely appreciate your valuable feedback, which has helped us improve the clarity and completeness of the manuscript. We also thank you for carefully considering our responses and raising your score. We will make sure to incorporate your suggestions into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.