acceptodds
Under review as a conference paper at ICLR 2027

ModelLakeFishing: Efficient Retrieval over Million-Scale Model Lakes

Abstract

Open model lakes may contain millions of reusable models, making it costly to identify suitable models for a new dataset. This setting requires learning indexable representations of models from heterogeneous evidence, including models, training or evaluation datasets, model cards, and historical evaluations. An important challenge is that this heterogeneous evidence is sparse and concentrated around popular models and benchmarks. We present ModelLakeFishing, a model-retrieval framework for queries that specify a target dataset, a prediction task, and an evaluation metric. It consolidates model lake metadata and historical evaluations into a model–dataset evidence graph, learns model and query embeddings with a structure-aware graph encoder, and indexes model embeddings with Hierarchical Navigable Small World (HNSW) search. At query time, HNSW retrieves 1,000 candidates without scoring every model in the lake. A task-level prior computed only from training-side evaluations then reranks the candidates for the requested metric and returns the top 10 models. Our evaluated lake We evaluate the framework on a lake containing 3,016,439 models and 247,803 model–dataset–task performance edges. In our evaluation, the models and datasets remain in the graph, while selected historical performance records are held out to test whether their best-performing models can be retrieved. Across three such evaluation splits, ModelLakeFishing achieves a mean eligible-query of 0.2968: the best model observed in the held-out records appears among the ten returned models for 29.68% of eligible queries. This retaisn 93.47% of the achieved by an exact baseline that applies the same combined scoring procedure to every model in the lake. Given precomputed query embeddings, candidate retrieval and reranking take 0.747ms at the median and 1.102ms at the 95th percentile. These results demonstrate the feasibility of efficient model retrieval across a multi-million-model lake under sparse supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Model Recommendation, Model Lake, Model Search, Model Retrieval, Supervised Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v9d9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ModelLakeFishing, a model retrieval framework for million-scale model lakes that aims to quickly generate candidate model lists for a target dataset and prediction task. The method integrates model and dataset metadata, historical evaluation records, dataset similarity, and model lineage into a heterogeneous evidence graph. It uses relation-aware GraphSAGE to learn shared vector representations, trained through three objectives: ranking, model contrast, and global competition. The authors construct a retrieval dataset from a model lake containing approximately 3 million models and 247000 historical performance records and evaluate their method on it.

Critical strengths
  1. The experiments provide a useful breakdown of retrieval quality. By comparing exhaustive ranking, exact retrieval followed by reranking, and approximate retrieval followed by reranking, the paper distinguishes the effect of candidate pool truncation from that of HNSW approximation.

  2. The method’s scalability is evaluated on a large collection of real models, demonstrating some engineering value.

Critical weaknesses
  1. The practical motivation is questionable. The paper is primarily motivated by the difficulty of selecting a model for a target dataset from a repository containing millions of models. However, repository size does not necessarily reflect the number of practically competitive candidates. These repositories contain many older models and community-fine-tuned variants of existing models. For users, a more direct approach would be to evaluate a small number of currently competitive open-source models or commercial LLMs. In particular, mainstream open-source LLMs are increasingly concentrated among a few providers, such as DeepSeek and Qwen, and their performance improves on a monthly basis. Models released several months earlier are generally no longer comparable to the latest models on most tasks. Therefore, directly selecting recent models with high download counts may be a more practical recommendation strategy.

  2. The paper does not adequately establish whether the improvement comes from the proposed graph model or primarily from the historical performance information added during reranking. Without the historical task prior, performance is approximately 12%, whereas adding it increases performance to approximately 33%, indicating that historical scores substantially influence the final results. Would directly recommending models based on their historical scores, or using a simple retrieval method followed by the same historical-score reranking, achieve similar results? The paper lacks these comparisons.

  3. The intended application is to select the best model for a given target dataset. However, during testing, the target dataset node already exists in the graph, and its name, description, and similarity relationships remain available; only its model performance records are hidden. Consequently, the evaluation tests whether the system can recover the historical best model from existing information. Whether the system can accurately recommend models for a previously unseen dataset uploaded by a user remains to be established.

Questions for authors
  1. Please correct the following grammatical and typographical errors:
  • Line 118: “for a tasks” → “for a task.”
  • Line 188: “dataset simialrity” → “dataset similarity.”
  • Line 190: Add a period after “…described in Section 3.2.”
  • Page 8, ablation table: “wo/ task prior” → “w/o task prior.”
  • End of the conclusion: Add a period after “…remains an important open experiment.”
  1. The experimental results table on page 8 lacks a table number and descriptive caption. Please add both.

  2. Why does removing the ranking loss improve performance? In the ablation table, removing this loss increases Gold@10 from approximately 33.20% to 35.03%.

Reviewer_6vwiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ModelLakeFishing, a framework for efficient model retrieval from large-scale model repositories. The key idea is to represent models and datasets as a heterogeneous graph, learn their representations using GraphSAGE, and perform efficient retrieval using an approximate nearest neighbor index. The framework also incorporates dataset descriptions and available performance information to identify models suitable for a given dataset. The experimental results suggest that the proposed framework can support efficient model retrieval at a large scale.

Critical strengths
  1. The paper addresses a practically important problem, as efficiently finding suitable pretrained models becomes increasingly challenging with the growing number of available models.
  2. The proposed framework demonstrates promising scalability and retrieval efficiency on a repository containing millions of models. This represents a meaningful practical contribution.
Critical weaknesses
  1. Despite its practical contribution, this work suffers from a lack of technical novelty. The proposed framework is largely built upon existing techniques, including heterogeneous graph representation, GraphSAGE, and HNSW-based approximate nearest neighbor search. While their integration provides a useful solution to a practical problem, it is difficult to identify a substantial new algorithmic or methodological contribution. In other words, the paper appears to contribute more as a system implementation than as a research advancement, which raises concerns about its significance for ICLR.

  2. The paper does not sufficiently establish its advantages over closely related model recommendation approaches. Prior work, such as TransferGraph and ModelLens, has already investigated model recommendation based on relationships between models and datasets. Although ModelLakeFishing emphasizes efficient retrieval at scale, the technical distinction from these approaches is not sufficiently supported by direct comparisons under comparable settings.

  3. The experimental results demonstrate scalability more convincingly than recommendation quality. Although the framework indexes approximately three million models, only a relatively small number of model–dataset performance records are available. Consequently, the experiments provide evidence that the framework can efficiently search a large model repository, but do not fully establish that it can reliably identify the best-performing models from such a large candidate set.

Questions for authors
  1. How does ModelLakeFishing compare with existing model recommendation approaches, such as TransferGraph and ModelLens, when evaluated on the same candidate models and under comparable computational budgets?
  2. How does the sparsity of observed model–dataset performance records affect the reliability of the retrieval results, particularly for models with little or no historical performance information?
Reviewer_bn7aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses efficient model discovery in large model repositories with sparse evaluation evidence. The proposed approach combines metadata, model lineage, and historical performance in a heterogeneous graph to learn model and query representations, followed by indexed retrieval and task-prior reranking. Experiments assess recovery of historically strong models and retrieval efficiency at million-model scale.

Critical strengths
  1. The paper considers the model recommendation problem with very large model pool, which receives few studies before. The graph-based method can be of practical usefulness in applications.
Critical weaknesses
  1. The soundness of the benchmark remains insufficient. The ground truth model-dataset performance is derived from sparse historical evaluations whose provenance and evaluation conditions are insufficiently documented. The designated gold model is only the best among recorded candidates, while potentially stronger, unevaluated recommendations receive no credit. Consequently, the benchmark measures recovery of historical winners rather than reliably establishing recommendation quality.

  2. The necessity of searching the entire heterogeneous model lake is not established. Task and modality filters could substantially reduce the candidate space, yet no filter-then-rank baseline is evaluated. Full-lake scale also serves to justify omitting relevant recommendation methods. A smaller, controlled benchmark with task-compatible candidates would enable stronger comparisons, with separate experiments assessing scalability.

Questions for authors

I would like to strongly recommend to include results under smaller scale benchmarks with relatively complete and sound ground-truth annotations. Performance comparisons with more existing model recommendation baselines are suggested under such tractable benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.