- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes
-
PMD Bench: a benchmark for external parametric memory routing. The authors organize the benchmarks into three categories, which are document-level knowledge, domain-level knowledge, and task-skill memory
-
Parametric Memory Decoding: the paper proposes zero-shot LoRA routing by measuring how each LoRA responds to a query using backbone activations and LoRA weights
-
PMDRouter: the authors develop a zero-shot routing method that scores each LoRA normalized response strength to a query
Critical strengths
-
The motivation is clear, and the methodology is logically aligned
-
The authors show careful consideration of various factors when formulating the routing framework and methodology.
-
The proposed method is relatively simple and intuitive
-
The authors provide comprehensive experiments and anlayses
Critical weaknesses
-
There may be a need for more clearer reason or theoretical reason why this stronger LoRA response based signal imply greater memory relevance. See question 1.
-
The authors mention the limitation of existing zero-shot routing methods is that they may struggle to distinguish LoRA memories trained on similar domains, related tasks, or overlapping documents. However, I cannot see how PMDRouter directly addresses this limitation. See question 2.
-
Generalization and robustness analysis is limited. I think I can ask some questions whether this is really generalizable and robust. See questions 3 and 4.
-
(Minor) Please be careful and ensure the consistency when writing terminologies especially capitalization. (ex1. Line 33 large language models, then you use Large language models. ex2. Line 40 Lora -> LoRA ) I stopped tracking them through out the paper but i can see some of them often.
Questions for authors
-
Why does a stronger LoRA response based signal imply greater memory relevance? Is there a theoretical explanation for this to always hold?
-
Can the authors clairfy or show additional experimetns or analysis comparing PMDRouter with existing baselines under different degrees of semantic similarity or overlapping knowledge? I cannot see why PMDRouter can be a solution to the limitations of existing zero-shot routing methods. I think this is critical
-
I am curious about how sensitive PMDRouter is to query variations. Specifically, if a query is paraphrased, would the router still select the same LoRA?
-
I am curious about how sensitive this method is to different LoRA configurations, such as rank, initialization. Specifically, for example, would LoRA adapters trained on similar knowledge but with different configurations produce comparable routing signals?
-
I am curious about what margin values would be considered sufficiently distinguishable or indistinguishable. Could the authors report the specific numerical ranges of the response margins and clarify how these ranges relate to routing accuracy? I think this would help us better understand how much separation is needed for reliable LoRA selection.
-
One reason to use zero shot routing is i think because of the cost. Could there be some concrete cost-related experiments or explanations that can show this?