acceptodds
Withdrawn from ICLR 2027

Parametric Memory Decoding for Zero-Shot Routing in LoRA-Based External Parametric Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

With the rise of parametric memory, LoRA-based External Parametric Memory (EPM) has emerged as a modular solution, but existing routing methods often introduce additional training, deployment, and maintenance overhead. This raises a natural question: can a LoRA-based EPM bank be routed without maintaining an additional routing component? However, existing zero-shot LoRA routing methods still face two problems under the EPM setting: (1) their evaluations are scattered across different task settings rather than organized around EPM access, and (2) their routing signals lack a unified perspective to guide systematic improvement. To address these problems, we organize PMD-Bench, covering document-level, domain-level knowledge, and task-skill, and propose Parametric Memory Decoding (PMD), the first framework designed to systematically improve zero-shot LoRA routing by reframing it as decoding activations over external parametric memory. Based on PMD, we further instantiate PMDRouter, which scores each LoRA by its response magnitude from a single base-model prefill. Experiments on PMD-Bench show that PMDRouter achieves the strongest internal-signal performance across multiple zero-shot routing settings, demonstrating the feasibility of zero-shot LoRA routing and the potential of PMD as a general framework for improving zero-shot routing methods. Sources: https://anonymous.4open.science/r/Parametric-Memory-Decoding-872A.

Primary area:
foundation or frontier models, including LLMs
Keywords:
External Parametric Memory, Zero-Shot LoRA Routing, Parametric Memory Decoding, Low-Rank Adaptation, Modular Large Language Models, Parameter-Free Routing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uRe8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes

  1. PMD Bench: a benchmark for external parametric memory routing. The authors organize the benchmarks into three categories, which are document-level knowledge, domain-level knowledge, and task-skill memory

  2. Parametric Memory Decoding: the paper proposes zero-shot LoRA routing by measuring how each LoRA responds to a query using backbone activations and LoRA weights

  3. PMDRouter: the authors develop a zero-shot routing method that scores each LoRA normalized response strength to a query

Critical strengths
  1. The motivation is clear, and the methodology is logically aligned

  2. The authors show careful consideration of various factors when formulating the routing framework and methodology.

  3. The proposed method is relatively simple and intuitive

  4. The authors provide comprehensive experiments and anlayses

Critical weaknesses
  1. There may be a need for more clearer reason or theoretical reason why this stronger LoRA response based signal imply greater memory relevance. See question 1.

  2. The authors mention the limitation of existing zero-shot routing methods is that they may struggle to distinguish LoRA memories trained on similar domains, related tasks, or overlapping documents. However, I cannot see how PMDRouter directly addresses this limitation. See question 2.

  3. Generalization and robustness analysis is limited. I think I can ask some questions whether this is really generalizable and robust. See questions 3 and 4.

  4. (Minor) Please be careful and ensure the consistency when writing terminologies especially capitalization. (ex1. Line 33 large language models, then you use Large language models. ex2. Line 40 Lora -> LoRA ) I stopped tracking them through out the paper but i can see some of them often.

Questions for authors
  1. Why does a stronger LoRA response based signal imply greater memory relevance? Is there a theoretical explanation for this to always hold?

  2. Can the authors clairfy or show additional experimetns or analysis comparing PMDRouter with existing baselines under different degrees of semantic similarity or overlapping knowledge? I cannot see why PMDRouter can be a solution to the limitations of existing zero-shot routing methods. I think this is critical

  3. I am curious about how sensitive PMDRouter is to query variations. Specifically, if a query is paraphrased, would the router still select the same LoRA?

  4. I am curious about how sensitive this method is to different LoRA configurations, such as rank, initialization. Specifically, for example, would LoRA adapters trained on similar knowledge but with different configurations produce comparable routing signals?

  5. I am curious about what margin values would be considered sufficiently distinguishable or indistinguishable. Could the authors report the specific numerical ranges of the response margins and clarify how these ranges relate to routing accuracy? I think this would help us better understand how much separation is needed for reliable LoRA selection.

  6. One reason to use zero shot routing is i think because of the cost. Could there be some concrete cost-related experiments or explanations that can show this?

Reviewer_HDPWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies the problem of how to route differently trained LoRA external memory modules. The paper proposes a benchmark (PMD-Bench) for this problem covering 3 different memory tasks and a method (PMD Router) that improves on zero-shot routing.

Critical strengths

The highly problematic writing and presentation of this paper makes it difficult to assess its strengths.

The observable strengths are that the paper compends a benchmark from various sources (Appendix A), providing a reasonable taxonomy of task types. Meanwhile, a novel method is proposed. While confidence interval is not provided, Table 1 suggests that the method's competitiveness over other zero-shot methods is plausible.

Critical weaknesses

The writing and presentation are a major issue in the paper.

The presentation and writing are confusing and make it very difficult to assess the problem the paper is tackling and the contribution of this paper. The recurring themes are not setting up the problem and abrupt introductions of highly specific terms without prior definition.

For example, in the abstract, the very first sentence: “With the rise of parametric memory, LoRA-based External Parametric Memory (EPM) has emerged as a modular solution, but existing routing methods often introduce additional training, deployment, and maintenance overhead.” The first sentence simultaneously introduces two very specific subfields, and when the reader is grounded at “parametric memory”, “routing methods” is suddenly brought up without prior definition - what are they routing? It took later sections and FIgure 1 for me to understand. The next sentence “This raises a natural question: can a LoRA-based EPM bank be routed without maintaining an additional routing component?” The paper needs to first introduce how routing is done and what is a main challenge in current routing, before bringing up “an additional routing component”. Then the abstract introduces the “problem”: “(1) their evaluations are scattered across different task settings rather than organized around EPM Access.” It is similarly confusing that both “scattered across different task settings” and “organized around EPM Access” appear in the same sentence like this. What is a “task setting”?

The key sections of the paper mentions terminologies without previously introducing or explaining them. For example, in the Contributions section, line 094, “Our contributions are threefold: (1) We organize PMD-Bench, the multi-granularity benchmark for external parametric memory routing. It covers document-level, domain-level knowledge, and task-skill memory with 35 memory-unit partitions in total (Section 3.2).” While words like "multi-granularity" and "memory-unit" indeed appeared in the introduction, they were not defined there either, making it impossible to understand what is meant by these words.

Figure 2 is too dense for viewing.

Similar writings and presentation issues as above occur throughout the paper, making the paper extremely hard to read and understand.

Confidence intervals are not provided in experimental results.

Questions for authors

See Weaknesses

Reviewer_iisxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies zero-shot routing for LoRA-based external parametric memory. It introduces PMD-Bench, covering document, domain, and task-skill memory, and proposes PMDRouter, which uses backbone activations and LoRA weights to select an adapter without a trained router. Experiments on three backbones show gains over other internal-signal methods, especially on Task-LoRA.

Critical strengths
  1. The problem is useful for modular LLMs.
  2. The experiments cover several memory settings and backbone models, with clear gains on Task-LoRA.
Critical weaknesses
  1. SpectR already uses a very similar routing signal. It is unclear how much normalization and calibration actually contribute. Fig. 5 also shows that the simpler projection-only decoder performs better than the full decoder.
  2. The zero-shot setting is questionable. PMDRouter still uses statistics from the training split for calibration, so adding a new LoRA seems to require access to its training data.
Questions for authors
  1. How much do normalization and calibration improve over SpectR when using the same pooling and layers?
  2. Why does the projection-only decoder outperform the full decoder on all three backbones in Figure 5?
  3. Does adding a new LoRA require access to its training data to compute calibration statistics? If so, how is this consistent with the zero-shot setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.