- Rating
- 1
- Confidence
- 4(+1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces the LoRAHunter framework, which leverages learnable probes, Transformer aggregation mechanisms, and contrastive learning to derive semantic representations directly from LoRA weights. Furthermore, it incorporates a reward mechanism based on generative tasks to optimize these representations, thereby facilitating "text-to-LoRA" retrieval and multi-adapter selection.
Critical strengths
NONE
Critical weaknesses
- The discussion or comparison with existing related work such as W2T, "A LoRA is Worth a Thousand Pictures," and WeightCLIP is insufficient.
- In retrieval tasks, the proposed method does not consistently outperform metadata-based baselines. A more thorough comparison with existing weight encoders is required.
- The generated results fail to clearly demonstrate whether the learned representations truly capture the actual utility of the LoRA adapters. The paper lacks a direct evaluation based on reward-driven ranking, nor does it assess the compatibility of multi-LoRA combinations.
- The set of baselines used in the experiments is too limited, particularly regarding weight-based retrieval. More competitive baselines should be introduced, and fair comparisons must be conducted under identical retrieval and combination settings.
Questions for authors
- How does LoRAHunter differ from existing research such as W2T ("A LoRA is Worth a Thousand Pictures") and WeightCLIP?
- Why does LoRAHunter not always outperform metadata-based baseline methods in retrieval tasks?
- How do the authors demonstrate that the learned representations truly capture the actual functional utility of the LoRA adapters?
- Could the authors include more competitive baseline methods specifically those based on weight parameters and evaluate them under the same experimental setup?
5 comments
Re W3 & Q3: How Do We Verify That the Learned Representations Capture the Functional Utility of LoRA Adapters?
For image generation, our primary concern is the quality of the generated outputs. We therefore evaluate LoRA utility directly through generation results. Table 4 in the main paper already shows that the retrieved LoRAs improve text alignment, measured by CLIPScore, and image quality, measured by PickScore and ImageReward. We further compare our retriever with direct retrieval using CLIP metadata embeddings. CLIP-global-Top-1 and Ours-global-Top-1 use CLIP embeddings and our learned weight embeddings, respectively, to retrieve a single LoRA using the full prompt for image generation. Ours-Concept-Top-1 retrieves the top-ranked LoRA for each concept using our weight embeddings and composes the retrieved adapters. Composition(Ours) denotes our full pipeline, while CLIP-Composition replaces our weight-based adapter representations with CLIP embeddings within the same pipeline.
| Method | prompt | CLIPScore ↑ | PickScore ↑ | ImageReward ↑ |
|---|---|---|---|---|
| Base | 100 | 26.656 | 22.412 | 0.353 |
| CLIP-global-Top-1 | 100 | 26.039 | 22.215 | 0.195 |
| Ours-global-Top-1 | 100 | 26.703 | 22.493 | 0.402 |
| Ours-Concept-Top-1 | 100 | 25.751 | 22.216 | 0.319 |
| CLIP-Composition | 100 | 26.518 | 22.441 | 0.308 |
| Composition(Ours) | 100 | 26.921 | 23.054 | 0.554 |
These empirical results indicate that the learned representations capture both semantic similarity and the functional utility of LoRA adapters. They also show that LoRAs retrieved solely through metadata similarity do not necessarily improve generation quality. LoRA quality varies considerably across the open-source LoRA ecosystem, and textual metadata alone provides limited insight into an adapter’s actual generation behavior. This motivates learning representations that incorporate generation feedback.
Regarding multi-LoRA compatibility, LoRAverse also discusses how composing multiple LoRAs can degrade image quality. We add an experiment that retrieves the top-ranked LoRA for each concept using our encoder and directly composes the retrieved adapters for generation. The results show that naively combining highly ranked LoRAs can introduce compatibility issues and reduce the quality of the generated images. These results have been added to Appendix A.2.6 (highlighted in blue).
Re W4 & Q4: Comparison with Other Weight-Based Methods
Appendix A.2.2 already includes a comparison with a weight-encoding baseline. We additionally implement W2T for comparison. W2T was originally trained with task-specific supervision, such as classification over a predefined set of labels. We adapt it to our setting by retaining its QR–SVD procedure and model architecture while training it with our contrastive learning objective. For a fair comparison, we retrain our model using the same data and training budget. Specifically, both models are trained on 4,000 LoRAs for 20 epochs, providing a controlled comparison of the two encoding approaches. We report retrieval and clustering results below:
| Method | Hit@1 | Hit@5 | Hit@10 | ARI ↑ | NMI ↑ | KNN@5 ↑ |
|---|---|---|---|---|---|---|
| W2T | 0.1167 | 0.4167 | 0.5167 | 0.0458 | 0.2218 | 0.1505 |
| Ours | 0.1667 | 0.4500 | 0.6667 | 0.1243 | 0.3316 | 0.2553 |
The results show that, under matched training budgets and experimental settings, our method outperforms both the structured weight-encoding baseline and the recent W2T method. Our method also has lower inference overhead. W2T requires matrix decompositions for each LoRA module, adding computational cost. Excluding disk I/O, encoding an SD LoRA takes approximately 0.75 seconds with W2T and 0.5 seconds with our method.These results have been added to Appendix A.2.8 (highlighted in blue).
Dear Reviewer s3Yq,
Thank you again for your comments. We are following up on our detailed response, where we clarified the differences from related work and added controlled experiments to isolate the contributions of weight representations and adapter selection, examine generation utility and composition, and compare additional weight encoders under matched training settings.
Our framework connects LoRA weights with natural-language queries and generation feedback. Its practical value is that users can retrieve relevant LoRAs even when metadata is missing or inaccurate. It also provides a basis for open-source platforms to assess adapter quality. More broadly, interpreting LoRA parameters directly may support future applications in adapter composition and capability customization.
We would appreciate it if you could review these additions and let us know whether they address your main concerns. We would be happy to clarify any remaining questions.
Best regards,
The Authors
We thank the reviewer for the constructive feedback and provide our point-by-point responses below.
Re W1 & Q1: Differences from Related Work
Thank you for highlighting these related works. LoRAHunter’s main advance is to connect community LoRA weights with natural-language requests and generation utility, allowing users to describe their desired output and retrieve and select suitable adapters. W2T learns weight representations through task-specific supervision, such as training a classification head to predict 40 predefined CelebA attributes. Its original setup requires additional text alignment to support direct natural-language LoRA search. LoRAHunter aligns weight embeddings with a frozen text encoder, supporting descriptions of objects, scenes, styles, and their combinations. For representation construction, W2T canonicalizes updates using QR–SVD, incurring additional matrix-decomposition overhead. Our learnable probes directly read low-rank updates, with their readout directions learned through semantic supervision. A LoRA is Worth a Thousand Pictures vectorizes LoRA factors and applies PCA with calibration to construct style descriptors, primarily studying style clustering and similarity-based LoRA retrieval. These descriptors do not directly express an adapter’s capabilities in natural language. LoRAHunter combines text alignment with paired generation feedback to learn an index that captures both semantic relevance and generation utility. WeightCLIP aligns full-model weights with dataset image samples, uses target image sets as queries, and evaluates primarily on CNN and ResNet-18 model zoos with a shared architecture. This requires access to corresponding image samples, while the original training data of community LoRAs are often unavailable. We train our encoder using community metadata as weak supervision and offline generation feedback. Once frozen, it can index new LoRAs for the same backbone directly from their weights. This integration of natural-language queries, weight-based indexing, and generation-oriented selection directly serves the needs of expanding community repositories of diffusion adapters. We have added a discussion of these works to the Appendix A.4 (highlighted in blue).
Re W2 & Q2: Why Does LoRAHunter Not Consistently Outperform Metadata-Based Baselines?
As explained in the experimental setup, the information sources used in this retrieval benchmark favor metadata-based methods. The category semantics expressed by the queries and the query–LoRA relevance annotations are derived from adapter metadata through manual annotation or LLM summarization. Metadata CLIP directly accesses titles, descriptions, and tags at retrieval time, including explicit semantic cues aligned with the evaluation target, whereas LoRAHunter must recover these semantics from adapter weights. The advantage of clean metadata on some ranking metrics is therefore consistent with this setup. Meanwhile, LoRAHunter performs better on other measures. Table 1 reports ARI of 0.810 versus 0.799, purity of 0.893 versus 0.876, cosine Silhouette of 0.229 versus 0.205, and an intra–inter similarity gap of 0.347 versus 0.201, comparing LoRAHunter with metadata CLIP. These results support the recovery of meaningful category structure from weights, with stronger performance on these clustering and separation measures. Combined with its ability to index new adapters directly from their weights, LoRAHunter provides a semantic retrieval signal that remains usable when metadata is missing, incomplete, or unreliable.