- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ModelLakeFishing, a model retrieval framework for million-scale model lakes that aims to quickly generate candidate model lists for a target dataset and prediction task. The method integrates model and dataset metadata, historical evaluation records, dataset similarity, and model lineage into a heterogeneous evidence graph. It uses relation-aware GraphSAGE to learn shared vector representations, trained through three objectives: ranking, model contrast, and global competition. The authors construct a retrieval dataset from a model lake containing approximately 3 million models and 247000 historical performance records and evaluate their method on it.
Critical strengths
-
The experiments provide a useful breakdown of retrieval quality. By comparing exhaustive ranking, exact retrieval followed by reranking, and approximate retrieval followed by reranking, the paper distinguishes the effect of candidate pool truncation from that of HNSW approximation.
-
The method’s scalability is evaluated on a large collection of real models, demonstrating some engineering value.
Critical weaknesses
-
The practical motivation is questionable. The paper is primarily motivated by the difficulty of selecting a model for a target dataset from a repository containing millions of models. However, repository size does not necessarily reflect the number of practically competitive candidates. These repositories contain many older models and community-fine-tuned variants of existing models. For users, a more direct approach would be to evaluate a small number of currently competitive open-source models or commercial LLMs. In particular, mainstream open-source LLMs are increasingly concentrated among a few providers, such as DeepSeek and Qwen, and their performance improves on a monthly basis. Models released several months earlier are generally no longer comparable to the latest models on most tasks. Therefore, directly selecting recent models with high download counts may be a more practical recommendation strategy.
-
The paper does not adequately establish whether the improvement comes from the proposed graph model or primarily from the historical performance information added during reranking. Without the historical task prior, performance is approximately 12%, whereas adding it increases performance to approximately 33%, indicating that historical scores substantially influence the final results. Would directly recommending models based on their historical scores, or using a simple retrieval method followed by the same historical-score reranking, achieve similar results? The paper lacks these comparisons.
-
The intended application is to select the best model for a given target dataset. However, during testing, the target dataset node already exists in the graph, and its name, description, and similarity relationships remain available; only its model performance records are hidden. Consequently, the evaluation tests whether the system can recover the historical best model from existing information. Whether the system can accurately recommend models for a previously unseen dataset uploaded by a user remains to be established.
Questions for authors
- Please correct the following grammatical and typographical errors:
- Line 118: “for a tasks” → “for a task.”
- Line 188: “dataset simialrity” → “dataset similarity.”
- Line 190: Add a period after “…described in Section 3.2.”
- Page 8, ablation table: “wo/ task prior” → “w/o task prior.”
- End of the conclusion: Add a period after “…remains an important open experiment.”
-
The experimental results table on page 8 lacks a table number and descriptive caption. Please add both.
-
Why does removing the ranking loss improve performance? In the ablation table, removing this loss increases Gold@10 from approximately 33.20% to 35.03%.