- Rating
- 1
- Confidence
- 3(−1)
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the problem of recovering “rank axes” in vision embedding spaces; rank axes are directions in the embedding space along which images can be arranged by known semantic attributes. While existing work recovers these axes using the linear regression loss w.r.t. ground truth values of the semantic attribute, this work studies the scenario where “soft ranks” are used as the ground truth, enabling one to employ differentiable objectives directly on the ranking information. Two of these objectives, a Spearman-type squared-rank loss and a Fenchel-Young loss are studied theoretically (w.r.t. their convergence and consistency) and empirically (w.r.t. their empirical performance against linear regression). I will refer to these two objectives as SRank-Sp and SRank-FY from here on out. With experiments using synthetic and real-world datasets, the paper aims to demonstrate that the proposed objectives (particularly, SRank-Sp) recover the underlying rank axes better than existing baselines.
Critical strengths
The paper compares different methods to recover rank axes from vision embeddings; especially with the added baselines suggested by me and the other reviewers, it might potentially be useful for future research.
Critical weaknesses
W1. Insufficient motivation for explicit ranking objectives
-
[Critical] The need for an explicit ranking objective is not sufficiently justified in lines 047-051. Granted that linear regression uses raw attribute values, isn't it a necessarily stricter objective that includes ranking (i.e., it’s not an imprecise proxy optimizing for something entirely different)? Recovering the numerical magnitudes correctly will necessarily recover the correct ranks. Therefore, I’m not convinced that including numerical magnitudes is a significant shortcoming of the baseline, thereby providing the core motivation for the paper.
-
[Critical] Continuing on the point above, one could also consider linear regression over the hard ranks themselves. The proposed methods add complexity in terms of implementation, and an extra hyperparamater . I therefore find it reasonable to expect that this additional complexity should be justified in terms of demonstrated superiority over a minimally adjusted baseline (in fact, the original rankability paper does use ranks for datasets where numerical magnitudes are not available). This stands beside my original point: the paper does not clearly establish why using numerical magnitudes is a shortcoming.
-
§4.1: This section is a good sanity check showing that the theoretical consistency and convergence proofs indeed carry over empirically. But the main observation here, i.e., the proposed ranking objectives converge and increasingly recover the original rank axis with more synthetic data, also follows rather directly from §3. It's nevertheless good to see that this happens under finite data, but that is also demonstrated by the real-data experiment in §4.3. Unless there is a deeper insight that I missed, I disagree with including it among the main contributions of the paper (line 023).
W2 (more critical than W1). Advantage of the proposed method over the linear regression baseline was not sufficiently demonstrated.
-
[Critical] §4.3: For the datasets where SRank has large advantages over the linear regression baseline, the reported numbers for the baseline are substantially smaller than those reported in the original rankability work (Sonthalia et al., 2025). Table 1 shows that SRank-Sp substantially outperforms linear regression on the three crowd counting datasets. Statistical strength notwithstanding, Sonthalia et al. report SRCCs of 0.854, 0.749, 0.887 on the three datasets respectively (see Table 2 in their paper), much closer to SRank’s reported numbers than they appear in Table 1 of this submission. Plausibly, this was caused by differences in the data split or hyperparameters, but I did not find discussion about it in the paper.
-
§4.3: error bars over multiple trials are missing from the results; as some differences are quite small (examples: UTKFace, Adience and KonIQ-10k in Table 1), it’s important to check whether the proposed method is still superior to linear regression after accounting for randomness.
-
§4.1: While my concerns regarding the significance of this section remain, the linear regression (OLS) baseline is quite critically needed here and is currently missing. This is crucial for contextualizing the convergence of the 5 methods shown here, 4 of which were introduced in the paper itself.
-
In §4.1, SRank-FY is claimed as the strongest method overall (line 376); §4.3 then shows that SRank-Sp usually outperforms SRank-FY. This somewhat limits the practical takeaways one may gain from the paper: i.e., which of the two should one use in practice? I find it limiting that the paper does not take a stand here.
Given these experimental flaws (W2), the current manuscript does not sufficiently validate its claims. Separately from this, I am not yet convinced of the significance of the research question as a standalone study (W1). I think addressing these concerns would require significant changes to the manuscript. Therefore, I recommend a clear rejection at this stage.
Questions for authors
Please treat my points under Critical Weaknesses as questions, particularly the first point under W2. In addition to these, I will consider the following towards a potential revision of my score.
Q1. The work builds substantially on top of the original rankability paper, Sonthalia et al. However, it does not evaluate some of the datasets used in that work ("Kinect", "HCI", and "AVA"). This brings the number of attributes considered in the paper, down to 3 from 7. Could the authors clarify why the above datasets were omitted (for instance, is the proposed method not compatible with them)?
Q2. Please specify the dataset splits used in the study, and the model selection procedure (hyperparameter search) conducted for the linear regression baseline. This is important for understanding the discrepancies between the original baseline numbers for rankability and those reported in the submission.
6 comments
Response to Reviewer Ebda (Part 2)
W2-1&Q2
-
Comparison with published results (Sonthalia et al., 2025): The cited values 0.854, 0.749, and 0.887 correspond to nonlinear MLP. The corresponding linear regression results are 0.843, 0.734, and 0.869.
-
Embeddings and dataset splits: We extracted the embeddings using the code released by Sonthalia et al. (2025). We obtained the datasets and split information from the sources specified in Table 1 of Sonthalia et al. (2025). For UTKFace, we use the train and test partition from the prior work cited there. For Adience, we follow the official fold assignments, while UCF-QNRF, ShanghaiTech-A/B, and KonIQ-10k use their official dataset partitions.
| Dataset | Train | Validation | Test |
|---|---|---|---|
| UTKFace | 11,833 | 1,314 | 3,287 |
| Adience | 13,847 | 1,432 | 3,343 |
| UCF-QNRF | 1,081 | 120 | 334 |
| ST-A | 270 | 30 | 182 |
| ST-B | 360 | 40 | 316 |
| KonIQ-10k | 7,058 | 1,000 | 2,015 |
-
Regression and model selection: Linear regression is trained on frozen embeddings using MSE and Adam with weight decay. We follow their hyperparameter search ranges and test 30 random configurations for each dataset and model. The learning rate is sampled log-uniformly from . Weight decay is sampled log-uniformly from . Horizontal flipping is randomly enabled or disabled. Each run uses 100 epochs and a batch size of 128 with cosine learning rate decay. The checkpoint within each run is selected by the highest validation SRCC. The regression baseline in the submission selected the final configuration among the 30 trials using validation MAE. In the revised experiments, we change this final selection criterion to validation SRCC. This improves the regression baseline primarily on
UCF-QNRFandST-A. -
Additional baselines: We additionally evaluate regression using log- and rank-transformed attributes (Log Ridge, Rank Ridge in the table, respectively) to assess sensitivity to the numerical scale of the attributes. We also include three additional methods suggested by other reviewers, namely RankNet, PiRank, and SmoothI. Both checkpoint selection within each run and final configuration selection for this baseline use validation SRCC. The table reports mean test SRCC across the same seven encoders for all methods. Raw (MAE) and Raw (SRCC) denote regression on the original attributes with final configurations selected by validation MAE and validation SRCC, respectively. For SRank-Sp and SRank-FY, we report mean (SD) across 20 random initializations, averaging SRCC over the seven embedding models in each run. (Given the limited time available to address comments from five reviewers, we were unable to conduct repeated experiments for the other methods in this particular comparison. We will include results from repeated runs in the revised manuscript.)
| Dataset | Raw (MAE) | Raw (SRCC) | Log Ridge | Rank Ridge | RankNet | ListNet | PiRank | SmoothI | SRank-Sp | SRank-FY |
|---|---|---|---|---|---|---|---|---|---|---|
| UTKFace | 0.7638 | 0.7630 | 0.7493 | 0.6736 | 0.6698 | 0.6908 | 0.6407 | 0.7169 | 0.7773(0.0005) | 0.7019(0.0100) |
| Adience | 0.9091 | 0.9091 | 0.9110 | 0.8664 | 0.8790 | 0.8813 | 0.4620 | 0.8790 | 0.9079(0.0007) | 0.9030(0.0009) |
| UCF-QNRF | 0.6575 | 0.6967 | 0.8104 | 0.7619 | 0.6917 | 0.2931 | 0.6185 | 0.6999 | 0.8603(0.0042) | 0.8693(0.0029) |
| ST-A | 0.4369 | 0.5084 | 0.6697 | 0.7200 | 0.6477 | 0.4878 | 0.6150 | 0.6519 | 0.7541(0.0171) | 0.7436(0.0159) |
| ST-B | 0.8025 | 0.8023 | 0.8553 | 0.7732 | 0.7204 | 0.6129 | 0.7544 | 0.7877 | 0.8829(0.0057) | 0.8873(0.0055) |
| KonIQ-10k | 0.7828 | 0.7832 | 0.7223 | 0.5715 | 0.6582 | 0.6793 | 0.7558 | 0.7114 | 0.7988(0.0004) | 0.7937(0.0001) |
Log-transforming the targets brings our regression results closer to the published values, although the remaining discrepancy has not been fully explained. SRank-Sp achieves higher mean SRCC than all regression baselines on every data except Adience. We will add the dataset split sizes to Table 3 and the regression training and selection details to Appendix A.1.2, and update the results to include SRCC-based model selection and log- and rank-target regression baselines.
Response to Reviewer Ebda (Part 3)
W2-2
We evaluated SRank-Sp and SRank-FY over 20 random initializations while keeping the data split fixed. The two tables below report test SRCC as mean (SD) for every dataset and embedding model.
[SRank-Sp]
| Dataset/Embedding | RN50 | ViTB32 | CNX | DINO-B14 | CLIP-RN50 | CLIP-ViTB32 | CLIP-CNX |
|---|---|---|---|---|---|---|---|
| UTKFace | 0.6653(0.0028) | 0.7660(0.0019) | 0.7913(0.0008) | 0.7739(0.0007) | 0.8025(0.0002) | 0.8145(0.0006) | 0.8280(0.0004) |
| Adience | 0.8299(0.0018) | 0.8894(0.0003) | 0.8983(0.0040) | 0.9163(0.0017) | 0.9313(0.0001) | 0.9442(0.0001) | 0.9460(0.0005) |
| UCF-QNRF | 0.8626(0.0086) | 0.8498(0.0116) | 0.8585(0.0093) | 0.8458(0.0098) | 0.8804(0.0104) | 0.8643(0.0076) | 0.8606(0.0108) |
| ST-A | 0.7563(0.0296) | 0.8109(0.0532) | 0.7809(0.0160) | 0.7006(0.0211) | 0.7452(0.0336) | 0.7410(0.0588) | 0.7436(0.0408) |
| ST-B | 0.9008(0.0067) | 0.8655(0.0319) | 0.8789(0.0040) | 0.8740(0.0027) | 0.8953(0.0093) | 0.8748(0.0161) | 0.8909(0.0065) |
| KonIQ-10k | 0.7566(0.0023) | 0.7499(0.0005) | 0.7787(0.0019) | 0.7454(0.0019) | 0.8475(0.0009) | 0.8357(0.0007) | 0.8777(0.0016) |
[SRank-FY]
| Dataset/Embedding | RN50 | ViTB32 | CNX | DINO-B14 | CLIP-RN50 | CLIP-ViTB32 | CLIP-CNX |
|---|---|---|---|---|---|---|---|
| UTKFace | 0.5539(0.0199) | 0.7467(0.0259) | 0.7518(0.0283) | 0.6098(0.0390) | 0.6934(0.0231) | 0.7888(0.0040) | 0.7690(0.0211) |
| Adience | 0.8228(0.0042) | 0.8894(0.0001) | 0.9038(<0.0001) | 0.9039(<0.0001) | 0.9177(0.0055) | 0.9374(<0.0001) | 0.9457(0.0006) |
| UCF-QNRF | 0.8711(0.0047) | 0.8714(0.0060) | 0.8625(0.0085) | 0.8573(0.0054) | 0.8744(0.0061) | 0.8792(0.0081) | 0.8695(0.0025) |
| ST-A | 0.7805(0.0370) | 0.7774(0.0270) | 0.7905(0.0220) | 0.7222(0.0281) | 0.7289(0.0612) | 0.6918(0.0418) | 0.7142(0.0906) |
| ST-B | 0.9013(0.0144) | 0.8833(0.0178) | 0.8758(0.0304) | 0.8640(0.0128) | 0.9012(0.0107) | 0.8934(0.0028) | 0.8922(0.0090) |
| KonIQ-10k | 0.7433(<0.0001) | 0.7507(0.0002) | 0.7744(<0.0001) | 0.7496(0.0003) | 0.8352(<0.0001) | 0.8263(0.0001) | 0.8764(<0.0001) |
Compared with the strongest regression baseline in W2-1&Q2, SRank-Sp has higher mean SRCC on UTKFace and KonIQ-10k. SRank-FY is higher only on KonIQ-10k. Neither method is higher on Adience. We will update Table 1 with the averaged results and Tables 4 and 5 with the detailed means and SDs.
Q1
The method can be applied to Kinect, HCI, and AVA. We omitted them from the initial evaluation for practical reasons.
- Kinect: Sonthalia et al. (2025) randomly held out six sequences for testing. We had not matched those sequence identities for direct comparison with their reported results.
- HCI: HCI provides five decade categories. We initially focused on datasets with more distinct label values. Our method is applicable to HCI using average rank labels.
- AVA: AVA was excluded from our experiments because processing approximately 230,000 training and validation images exceeded the memory constraints of our current implementation, precluding a comparison with the other methods.
Response to Reviewer Ebda (Part 1)
Thank you for your careful review. Please see our responses to the concerns below.
W1-1
We agree that exact recovery of attribute values necessarily preserves their ordering. Our concern is not that linear regression ignores ranking, but that minimizing squared prediction error need not yield the direction that best preserves the ordering when the attribute values cannot be represented exactly by a linear projection.
In particular, rankability depends only on the ordering of attribute values and is invariant to strictly increasing transformations of those values. Such transformations preserve the set of order-preserving axes but can change the least squares regression direction. Thus, regression imposes additional requirements on the numerical magnitudes and spacing that are not necessary for rank axis recovery.
Our motivation is therefore to align the estimation objective directly with the definition of rankability, rather than to suggest that numerical supervision is inherently disadvantageous. We will revise the Introduction to clarify this distinction.
W1-2
We agree that linear regression on hard ranks is a natural and important baseline, as it removes dependence on the original attribute magnitudes without introducing a differentiable ranking operator.
The key distinction is that rank regression minimizes the squared difference between projected scores and numerical rank targets, whereas SRank-Sp compares the soft ranks of the projected scores with the target ranks. Rankability requires agreement in ordering, not agreement in numerical spacing. Under quadratic regularization, soft ranks coincide with hard ranks when all adjacent projected score gaps are at least ln (Blondel et al., 2020, Appendix B.3). Consequently, correctly ordered and sufficiently separated scores can achieve zero SRank-Sp loss without matching the numerical rank values. The same condition also yields zero SRank-FY loss.
To determine whether this distinction leads to a practical advantage, we have added linear regression on hard rank targets as an additional baseline, following the same experimental protocol. The results show that the proposed ranking approach retains its advantage over this baseline, indicating that applying a rank transformation to the predicted scores provides benefits beyond transforming the targets alone. We will include hard rank regression and the other additional baselines in the revised manuscript. Please see the table in our response to W2-1&Q2 for the full comparison.
W1-3
We agree that the synthetic experiments primarily serve to validate the theoretical findings under controlled finite-sample settings, rather than constitute an independent methodological contribution. Their main purpose is to evaluate recovery of a known ground-truth axis under varying sample sizes, dimensions, and noise levels, which cannot be directly assessed on real-world datasets. We will therefore revise the contribution statement to avoid presenting the expected synthetic recovery behavior as a separate novel finding. Instead, we emphasize the statistical recovery analysis and the empirical improvements observed on real-world ranking tasks, while retaining the synthetic experiments as supporting evidence.