- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RADAR, a pipeline for augmenting unit normalized embeddings by changing their norm, thus leading to altered ranking in retrieval settings. Using this method, the norm of the embeddings can therefore be adapted to encode an additional property without retraining the original embedding model, thus allowing for altered ranking behavior while retaining the underlying angular similarity structure. The framework thereby allows for using multiple auxiliary proxies for the desired property that shall be encoded by the vector norm and creates a scheme that predicts a robust value from these individual auxiliary predictors (with "robust" meaning a single feature that is reliably predictable from the embedding while staying faithful to the aspect that the proxies encode).
Experiments are conducted in the scientific domain, so that the norm is in practice adapted to encode the importance of a paper, as indicated by three auxiliary predictors: citations received one, two and three years after publication, and the standing of the venue the paper is published at. The underlying assumption is that a paper with high citation count or a paper that was published at a better venue should rank higher in a retrieval setting since it likely contains a fundamental contribution instead of a minor incremental change.
The individual predictors are used to create a more robust feature vector for each training example, by not encoding values directly, but instead relying on ranks across comparable papers (i.e., published in the same year) and additional normalization. Using these feature vectors created for training samples, the authors then train an additional model that predicts these feature vectors directly from an embedding created by an existing embedding model, which therefore makes this method applicable to any embedding at inference time (without needing actual citation counts or venue information).
Multiple features are then mapped onto a single scalar via a procedure that optimizes for maintaining most of the available variance of the three-dimensional feature vectors while focusing on a subspace that can be reliably predicted the given embedding by the neural network predictor. An extensive formal analysis demonstrates that this specific setup is beneficial, since 1) it bounds the percentage of explained variance of the trained predictor to be close to that of the optimal predictor 2) it reliably selects a direction close to the first pricipal component if the others do not encode sufficient variance, and 3) since the hyperparameter gamma (which controls the strength of the norm change) can be used to bound the maximal change in ranking that can occur.
The proposed method is then evaluated on multiple benchmarks, applied to a Qwen3-Embedding-8B model. Multiple existing models are included as baselines, including other strategies that also just adapt the norm of the predicted embeddings. It is shown that the proposed method outperforms all other methods on the benchmarks that are evaluated. Additionally, ablations are performed for several parts of the pipeline and an unrelated benchmark task from the SciRepEval benchmark is also evaluated.
To already preempt the strengths and weaknesses: This paper seems to generally propose a very sensible method that has good theoretical justification and demonstrates improved performance on the specific benchmarks that were evaluated. However, the evaluation fails to demonstrate the the method has any real benefit for other task types that are not as closely related to predicting highly cited papers, and does therefore not prove that the method is useful for any common real-world task. Ultimately, I therefore feel like this paper is precisely on the edge between a weak accept and a weak reject, with the final decision being the positive one due to the sound derivation and justification of the method.
Critical strengths
- The proposed method seems like a reasonable strategy to use incorporate these auxiliary signals into alterations of the embedding norm. One specific benefit thereby is especially the robustness of the feature preprocessing (i.e., using ranks instead of values and normalizing at different parts of the pipeline).
- A major contribution is also the optimization of the vector v that is used as weight on the individual auxiliary features to combine them into a scalar. The proposed scheme optimizes it to focus on the subspace of the feature space (i.e., of the multi-dimensional feature vector) that is both well predictable and encodes much of the variance encoded in the features. this makes an inherent trade-off visible and adresses it via a sensible optimization strategy.
- The theoretical analyses provide good support for the underlying strategy, proving that the specific optimization that optimizes for retaining a lot of feature variance while also encouraging good predictability has likely positive effects on the quality of the representation (exact effects can not be established though due to requiring knowledge of the optimal predictor of feature vector Y given embedding U). The additional analysis on the preservation of ranking for specific choices of the hyperparameter gamma then underscores the main argument of just reranking while keeping most of the structure intact.
- The evaluations include a good set of baselines, including comparable methods that also just alter the embedding norm, and evaluating the proposed scheme across multiple embedding backbones. This therefore demonstrates that the proposed method leads to small gains on the four specific benchmark tasks that were evaluated. The ablations then also show very well that several parts of the pipeline are crucial in achieving good performance.
Critical weaknesses
- The main weakness of this work is the limited scope of the evaluation. The underlying method is presented as a way of injecting useful information into existing embeddings, which is supposed to lead to generally improved embeddings. However, the evaluation does only establish that injecting the specific information does improve performance on exactly that type of task that benefits specifically from the injected information, thus not proving generally improved performance. The reason for this is that the paper evaluates just four main benchmark tasks and that all of them are strictly related to either predicting whether one paper should cite another, or to predict google scholar rankings, which likely ranks highly cited papers higher than others. This is a biased evaluation setup since the proposed method gives higher weight to generally highly cited papers, which of course are also more likely to be a valid target in all of these benchmarks (since highly cited papers are simply more likely to either be real citations or recommended citations, which is what these benchmarks rely on for construction). This becomes even more limiting considering that these specific tasks are usually not tasks that an embedding model is used for in practice. Usually, an embedding model is used to find highly relevant results for search queries (which are not (!) supposed to be just the highly cited foundational works but actually the most relevant results for the specific query), for stuff like RAG (which also needs to most precise matches and not just the most popular ones), or for semantic visualization, which generally does not benefit from few papers that have generally increased similarity to all of their neighbors due to a larger embedding norm. The existing evaluation therefore does not establish that the resulting model is useful in any real downstream application.
- This becomes even more suspect considering that exactly one task of the SciRepEval benchmark is used for a further small-scale evaluation, which is limited to just those methods that alter the embedding norm of the Qwen model and not the other standard baselines. Why not include the other baselines or the unlatered Qwen model in this evaluation? As it is now, we do not know how any of these approaches performs compared to normal scientific embedding models on this task. More interestingly, the SciRepEval benchmark comprises 24 tasks in total, so why just select one of them and not just include a complete evaluation of all baselines on all tasks of the benchmark? Note that I am aware that this benchmark has several issues. However, including the results from this benchmark can still give valuable insights into the properties of the different methods, and omitting them simply leads to a state in which this paper fails to demonstrate broader usefulness of the proposed method.
- The ablations show that each part of the pipeline is crucial for achieving the performance that outperforms the baselines. However, most of the baselines that perform a similar strategy of altering the embedding norm of the frozen embedding backbone seem to operate just on one of the proxy signals, and it is not clear if any of them had access to all three that were used for RADAR. Since dropping any of them makes RADAR not outperform the other baselines any more, this questions if the general training scheme is really responsible for the improved performance, or just the combination of multiple proxies for paper importance.
Questions for authors
Do any of the baselines use all three proxies? Why did you not evaluate on more general benchmark tasks?
3 comments
Response to Reviewer rjM8 (2/2)
W2: We thank the reviewer for this constructive comment. We would like to clarify that the Peer Review Score task in Section 5.4 is used to validate the learned latent impact score, not to evaluate retrieval. It measures how well the predicted impact score agrees with peer-review scores (Table 3), so only methods that predict such a score can be compared.
We agree that a broader SciRepEval evaluation is valuable. Following the reviewer's suggestion, we have added five subtasks of the SciRepEval benchmark, covering ad-hoc search (TREC-COVID and Search) and proximity (RELISH, CoView, and CoRead), with standard baselines, including Qwen3-Embedding-8B and scientific embedding models (Appendix E.5). We focus on ranking tasks because they directly evaluate retrieval quality, which is how RADAR is used. The main results (nDCG@10, %) are shown below, and results for all baselines are reported in Table 10 of Appendix E.5.
| Method | TREC-COVID | Search | RELISH | CoView | CoRead | Avg. |
|---|---|---|---|---|---|---|
| SPECTER2 | 72.04 | 79.84 | 67.49 | 87.67 | 89.16 | 79.24 |
| Qwen3-Embedding-8B | 86.29 | 84.63 | 74.44 | 89.46 | 91.90 | 85.34 |
| Qwen3-Reranker-8B | 87.09 | 85.64 | 74.91 | 90.11 | 91.68 | 85.89 |
| Impact-Aware Methods (Qwen3-Embedding-8B) | ||||||
| +MultiProxyPred | 86.82$_{\pm0.20}$ | 86.13$_{\pm0.24}$ | 76.07$_{\pm0.13}$ | 90.22$_{\pm0.16}$ | 92.18$_{\pm0.10}$ | 86.28 |
| +RADAR (Ours) | 86.16$_{\pm0.18}$ | 86.40 | 76.91 | 90.25 | 92.53 | 86.45 |
RADAR achieves the highest average nDCG@10, and significantly improves over the backbone on four of the five tasks after Holm correction, including Search (+1.77) (Table 11 of Appendix E.5). On TREC-COVID the difference from the backbone is within noise (−0.13, not significant), consistent with the bounded ranking perturbation in Proposition 4.
W3: We thank the reviewer for this important comment. In fact, two baselines use all proxies. CitationPred and VenuePred each use one proxy family, whereas MultiProxyPred and FactorDistill use the same citation and venue proxies as RADAR. MultiProxyPred averages the citation-based and venue-based predictors, and FactorDistill uses a classical linear factor model. In addition, two ablations in Table 2 (− Text Predictability and − Variance Retention) keep all proxies and only change the PALIF objective. These methods therefore use the same information as RADAR. Their macro-averaged results (%) on the four benchmarks are:
| Method (all proxies) | nDCG@10 | Recall@50 |
|---|---|---|
| Impact-Aware Methods (Qwen3-Embedding-8B) | ||
| +MultiProxyPred | 45.42$_{\pm0.15}$ | 50.41$_{\pm0.11}$ |
| +FactorDistill | 45.27 | 50.38 |
| RADAR − Text Predictability | 45.13 | 50.35 |
| RADAR − Variance Retention | 44.90 | 50.12 |
| RADAR | 46.02 | 51.31 |
With the same proxies, all these methods fall below RADAR. RADAR's gains over the strongest impact-aware baseline for each metric (+0.59 nDCG@10 over CitationPred and +0.90 Recall@50 over MultiProxyPred) are also significant after Holm correction (Table 6 of Appendix E.1). Removing a proxy family (− Citation Proxies or − Venue Proxies in Table 2) reduces the information available to RADAR, so a performance drop is expected. When the information is the same, the results suggest that using multiple proxies alone is not sufficient, and that the way PALIF combines the proxies is important for the gains.
Q1: Yes. MultiProxyPred and FactorDistill use the same citation and venue proxies as RADAR. The two ablations in Table 2 (− Text Predictability and − Variance Retention) also use all proxies.
Q2: Most general retrieval benchmarks provide only relevance labels. They do not indicate which relevant documents are also important, so they cannot directly evaluate the goal of RADAR, which is to rank important documents higher among relevant ones. This is why we constructed SuReIR. For example, in SuReIR, a paper cited in a survey is treated as relevant, and it receives a higher grade if the survey describes it as foundational (e.g., "seminal" or "first proposed") or cites it repeatedly. We believe such labels better reflect papers that are both relevant and important. To further examine general retrieval, we have also evaluated RADAR on five SciRepEval ranking tasks and on two cross-domain datasets (WANDS and CQADupStack), where RADAR achieves the best average results (Appendices E.5 and E.6).
We hope these responses address the reviewer's concerns, and we are happy to discuss any remaining questions.
Thank you for the extensive response.
Regarding W1, if your goal is to uprank papers with high influence, then this can be simply done by scaling the norm by the number of citations (or some transform of it, like the log or something), and does not require such a complex scheme, especially since 1) papers published at major venues can have little influence, and arxiv papers are sometimes some of the most influential ones and 2) citation counts are easily accessible for pretty much all papers, thus replacing the need for predicting them with a neural network.
For this exact reason, I personally feel like the main contribution of your work is not simply the application to scientific retrieval (since otherwise the contribution would not be sufficient), but it instead is the general scheme of using different factors (that might not be present at inference time) and combining them in a robust way. The evaluation on the scientific domain does prove that this method then has one specific benefit, which is rather limited for proving that the general method makes sense. As a result, I feel like weakness 1 still stands.
Regarding point 1.2, I did not mean to imply that you encode the exact signal that is evaluated. But you encode a signal that certainly is highly correlated with what the tasks demand, and this weakens the proof that this method is generally of benefit. You say that other methods like CitationPred do not show the same behavior. While this might be true, analyzing this myself as a reviewer is certainly out of the scope of what is required, and if this point is a major argument that disproves my high-level concern, then it might be a good choice to include a deeper analysis of all this in the paper, so that a reviewer does not need to come up with it themselves.
For the two additional tasks that you evaluate, it is not clear to me how they fit the general framework. For duplicate question search, for example, why would it be an inherent benefit to encode the rating of the question? If two questions are duplicates of each other, then what does the rating of them have to do with that?
Regarding W2, I honestly feel like selectively evaluating on a subset of benchmark tasks is simply a suspicious choice. I explicitly do not want to imply that you do this with bad intent since you likely made that decision in good faith, but as a reader you simply can not be sure if the tasks were simply selected due to your model performing well on them.
If you think that just a specific set of tasks is relevant for your model, then it would be best to nevertheless evaluate it on all tasks, and then discuss the benefits of your model on the important subset to prove your claims.
So overall I support the addition, but it should use all tasks from the benchmark.
Ultimately, the response and the reported additions slightly improve this work, but since I feel like my original assessment of this work was rather generous originally, I will leave my score as it is.
Response to Reviewer rjM8 (1/2)
We greatly appreciate the reviewer's detailed, specific, and constructive comments. We address each of the reviewer's comments and questions below.
W1: We agree that citation-based benchmarks may partly favor influential papers. However, we would like to clarify three points. (1) Scholarly importance is part of what we aim to capture. For scientific literature retrieval, users often want not only relevant papers but also important or foundational ones. Our goal is to improve retrieval by combining both types of information. (2) Our benchmarks do not directly use our training proxies as relevance labels. We intentionally avoided this when designing the experiments. SuReIR measures a paper's importance within a specific survey, rather than its global citation count or venue ranking. GSRR uses Google Scholar results, which may reflect prominence, as the reviewer correctly points out. (3) Our improvements are not limited to these two new benchmarks. RADAR also improves on MasterSet and RMC, two independently developed benchmarks (+0.52 and +1.16 nDCG@10, respectively). Interestingly, GSRR shows the smallest improvement (+0.19), suggesting that the gains are not simply driven by Google Scholar's preference for prominent papers.
On SuReIR, all impact-aware baselines, including CitationPred, perform below the backbone (−0.05 to −0.61 nDCG@10), whereas RADAR improves it by 1.58 points. This suggests that simply favoring papers with high predicted citations is not sufficient to reproduce RADAR's gains on SuReIR.
In addition, the influence of the learned attribute is bounded. For documents with positive cosine similarity, RADAR can reverse the order of two documents only when the ratio of their semantic scores is below eγ, which is about 1.05 at γ=0.05 (Proposition 4). Therefore, a highly cited paper is not ranked above one with a clearly higher semantic score.
We also agree that it is important to show benefits in downstream applications. To evaluate the generalization of RADAR beyond scientific literature, we have conducted additional cross-domain experiments on product search (WANDS) and duplicate-question retrieval (CQADupStack) in Appendix E.6. WANDS uses human relevance annotations, and CQADupStack uses duplicate questions marked by Stack Exchange users. RADAR learns a separate attribute in each domain from domain-specific proxies, while the backbone remains frozen. The main results (%) are shown below, and results for all baselines are reported in Table 13 of Appendix E.6.
RADAR achieves the best results on both datasets. Compared with the backbone, it improves nDCG@10 and Recall@50 by 1.91 and 1.50 points on WANDS, and by 2.58 and 2.86 points on CQADupStack. All four gains are significant after Holm correction (Table 14 of Appendix E.6).
Regarding visualization, RADAR only rescales each embedding by a factor between 1 and eγ and does not change its direction. The original embedding can be recovered by normalization, so visualizations based on the original embeddings or on cosine similarity are not affected.