acceptodds
Under review as a conference paper at ICLR 2027

Fast and Accurate: Improving Personalized LLM Performance by Semantic Concentration Inspired Workload Split between RAG and Fine-Tuning

Abstract

With ever increasing personalized large language model (LLM) services, the next question to answer is how to improve the quality of services/users' experiences, which is indicated by personalized LLM generation accuracy/quality and inference latency. Although state-of-the-art designs, e.g., parameter-efficient fine-tuning (PEFT), retrieval-augmented generation (RAG) or Full PEFT+RAG, can help, they largely ignore the heterogeneous semantic concentrations of users' historical data and cannot fully unleash the potential of personal data for personalized LLM performance enhancement. In this paper, we propose PEFT Union RAG (PUR), a novel "fast and accurate” approach that exploits the semantic concentration of users' personal historical data to split workload between PEFT and RAG, aiming to improve personalized LLM performance in terms of better generation accuracy/quality and lower inference latency. Briefly, PUR partitions user's personal historical data and assigns semantic concentrated "core” information to PEFT for offline parametric fine-tuning, while retaining dispersed information for RAG. During the online LLM inference, PUR directly generates responses for "core” queries covered by the PEFT region and invokes RAG only for sporadic "peripheral” queries. Experimental results show that, by appropriately splitting workload between PEFT and RAG, PUR consistently achieves the best generation accuracy/quality for various users across multiple tasks, while reducing inference latency by 14.4%–46.6% compared with the best performed method.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Personalization, Retrieval-Augmented Generation, Parameter-Efficient Fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7AEiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a fast and accurate method to improve personalized LLM performance, named PEFT Union RAG (PUR). It splits workload between PEFT and RAG according to their semantic concentration, aiming to enhance the performance of personalized LLMs with better generation accuracy/quality and lower inference latency. Records close to a semantic center are used for fine-tuning, while the remaining records form the retrieval corpus. During the online inference phase, if a user's query falls within the core semantic region covered by PEFT, the model generates a response directly; otherwise, RAG is invoked to retrieve external knowledge.

Critical strengths
  1. The paper addresses a practically important problem. Personalized models must exploit user histories to improve generation quality while maintaining acceptable inference latency, making the joint consideration of quality and latency highly valuable.
  2. PUR uses the distribution of user histories to determine both the fine-tuning subset and the retrieval path, providing an intuitive approach to coordinating coordinating parametric fine-tuning and retrieval augmentation.
  3. Experimental results show consistent improvements of PUR on better generation accuracy and shorter inference latency for various users across multiple tasks.
Critical weaknesses
  1. PUR maps the semantic concentration metric SconcS_{conc} to the PEFT data allocation ratio through an empirically calibrated look-up table (Table 2), rather than learning the allocation adaptively. Its applicability to new domains or users with substantially different historical data distributions requires additional justifications.
  2. Users with very limited historical data may not provide enough samples to estimate a reliable semantic center, and such unstable estimates could produce unreliable inference-path selection or lead to an excessive reliance on RAG. Have the authors considered the robustness of the PUR framework under these low-data conditions?
  3. Recent research has already explored personalized LLMs utilizing a full combination approach (100% PEFT + 100% RAG). The paper should more clearly distinguish the contribution of PUR and compare against representative existing methods to justify the proposed mechanism.
  4. The experiments only use Llama-2-7B as the base LLM, leaving it unclear whether this semantic concentration-based allocation strategy remains effective when scaled to larger LLMs or other SOTA models with stronger in-context learning capabilities.
Questions for authors

Please refer to the weakness above

3 comments
AuthorsOpenReview ↗

Author's Response

Thank you for your valuable comments! Please find below the responses to each comment on weaknesses.

[W1] PUR maps the semantic concentration metric to the PEFT data allocation ratio through an empirically calibrated look-up table (Table 2), rather than learning the allocation adaptively. Its applicability to new domains or users with substantially different historical data distributions requires additional justifications.

Response: We agree that new domains with substantially different historical data distributions may require recalibration of the mapping between semantic concentration scores and allocation ratios. However, this only introduces a one-time offline calibration cost for new domains. The core contribution of our work is to demonstrate the benefits of allocating workloads between PEFT and RAG in improving task performance and reducing online inference latency for users with different historical data distributions. Such recalibration only concerns the score-to-ratio mapping and does not change PUR's advantages in reducing online inference latency and improving task performance.

[W2] Users with very limited historical data may not provide enough samples to estimate a reliable semantic center, and such unstable estimates could produce unreliable inference-path selection or lead to an excessive reliance on RAG. Have the authors considered the robustness of the PUR framework under these low-data conditions?

Response: Thank you for raising this concern. We agree that PUR may be less reliable under extremely limited historical data conditions. However, such extreme data scarcity also poses challenges for standard PEFT and RAG methods. We have already investigated the impact of historical data size on PUR's performance in Figure 4. The results show that PUR consistently outperforms the compared methods across different semantic concentration levels and historical data sizes, even in settings with as few as 40–50 historical records. These results provide empirical evidence that PUR can maintain its performance advantage under relatively limited historical data conditions.

[W3] Recent research has already explored personalized LLMs utilizing a full combination approach (100% PEFT + 100% RAG). The paper should more clearly distinguish the contribution of PUR and compare against representative existing methods to justify the proposed mechanism.

Response: We would like to clarify that the core contribution of PUR is a novel semantic concentration-based workload allocation strategy between PEFT and RAG (e.g., 60% PEFT and 40% RAG according to semantic concentration). We never claimed that we proposed the full combination (100% PEFT + 100% RAG) as a new approach. Actually, we discussed SOTA full combination methods [1-2] in Section 2. Moreover, we have explicitly included Full PEFT+RAG as a compared method in Section 5.

[1] Z. Tan et al. Democratizing Large Language Models via Personalized Parameter-Efficient Fine-Tuning. EMNLP, 2024.

[2] B. Pingua et al. Medical LLMs: Fine-Tuning vs. Retrieval-Augmented Generation. Bioengineering, 2025.

[W4] The experiments only use Llama-2-7B as the base LLM, leaving it unclear whether this semantic concentration-based allocation strategy remains effective when scaled to larger LLMs or other SOTA models with stronger in-context learning capabilities.

Response: Thank you for this comment. We conduct additional experiments using Llama-2-13B and Llama-3.1-8B as base models, following the same experimental settings as Table 3. The results on the Personalized Scholarly Title Generation task are reported below.

Base LLMMethodUser1 (Sconc:0.701) Overall Perf. (R-1) ↑User1 (Sconc:0.701) Avg. Infer. Time(s)↓User 2 (Sconc:0.743) Overall Perf. (R-1) ↑User 2 (Sconc:0.743) Avg. Infer. Time(s)↓User3 (Sconc:0.809) Overall Perf. (R-1)↑User3 (Sconc:0.809) Avg. Infer. Time(s)↓
Llama-2-13BPEFT48.82.3148.62.5649.72.15
Llama-2-13BRAG51.74.5351.24.2252.04.07
Llama-2-13BFull PEFT+RAG51.44.3051.04.7952.93.98
Llama-2-13BPUR (Ours)53.23.7752.83.1954.12.48
Llama 3.1-8BPEFT49.71.3950.51.4952.21.38
Llama 3.1-8BRAG51.62.8752.32.9152.02.74
Llama 3.1-8BFull PEFT+RAG50.52.7652.63.1052.82.95
Llama 3.1-8BPUR (Ours)52.92.1353.12.1153.91.97

The results show that PUR consistently achieves the best task performance across all three users under both Llama-2-13B and Llama-3.1-8B, while reducing average inference latency compared with the retrieval-based methods. These results demonstrate the effectiveness of our semantic concentration-based workload allocation strategy across different LLMs.

I appreciate the authors' efforts. My major concerns have been addressed. I raise my score from 2 to 3.

AuthorsOpenReview ↗

Thank You and Rating Update

Dear Reviewer,

Thank you very much for your positive feedback and for reconsidering your assessment. We truly appreciate your time and support.

We noticed that the rating in the Official Review still shows 2. If possible, could you kindly update it to reflect your revised score of 3?

Thank you again for your consideration.

Best regards,

The Authors

Reviewer_jD1ZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes PEFT Union RAG (PUR). At inference time, queries close to the learned semantic regions bypass retrieval. The authors evaluate PUR on six LaMP tasks and report improvements in generation quality and inference latency over PEFT, RAG, and their full combination.

Critical strengths
  • The motivation is pretty reasonable and strong. The observation that PEFT and RAG perform differently depending on the semantic concentration of user histories is interesting. It gives a concrete reason to consider different adaptation strategies for different users.
  • The experiments cover multiple personalization tasks, including classification and text generation.
  • The method is relatively simple to implement using existing PEFT and retrieval components.
Critical weaknesses
  • The experiments do not sufficiently isolate the contribution of semantic concentration. PUR changes several things at once: which samples are used for PEFT, which samples remain available for retrieval, and whether retrieval is performed for a given query. The baselines in the paper do not separate these choices. For example, a fixed-ratio data split, a random split with the same training budget, or a PEFT+RAG baseline with selective retrieval would provide more useful comparisons. In particular, some of the latency improvement may simply come from skipping retrieval rather than from the proposed concentration-based allocation. The analysis in Figure 5 supports the concentration score's relationship with the selected ratio, but does not directly show how much the long-tail compensation term improves generation quality.
  • The inference routing rule needs stronger justification. In Section 4.3, a query is assigned to the PEFT path when its similarity to a cluster center exceeds the threshold obtained from the selected training samples. However, semantic similarity alone does not establish whether the fine-tuned model has learned the information needed to answer that query correctly.
  • The writing feels somewhat amateurish and lacks the level of academic rigor expected for a conference paper. For example, the abstract repeatedly uses quotation marks around expressions such as "fast and accurate", "core", and "peripheral". I find the use of quotation marks around "fast and accurate" particularly odd, as it makes the phrase sound more like a marketing slogan than a precise technical claim. Similar issues appear throughout the paper. For instance, the introduction states "It is obvious that the benefits brought by RAG come at the cost of severely delayed LLM response", which sounds informal. There are also awkward expressions such as "the best performed compared method" in the conclusion. I would encourage the authors to revise the manuscript more carefully and adopt a more precise academic writing style.
Questions for authors
  • How reliable is the inference routing decision? For queries assigned to the PEFT region, how often would retrieval have produced a better response?
  • In Appendix A.2, the authors reuse allocation ratios obtained with BGE when evaluating E5 and GTE, even though the semantic concentration scores change. How would PUR perform if the allocation ratios were determined directly from the new embedding models? Would the method require recalibration for each embedding model?
4 comments
AuthorsOpenReview ↗

Author's Response (1)

Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment on weaknesses and questions.

[W1.1] The experiments do not sufficiently isolate the contribution of semantic concentration.

Response: To isolate the contribution of semantic concentration-based data selection, we conduct a controlled comparison with equal-size random subsets. Specifically, to exclude the influence of inference routing, both methods use the same amount of data for PEFT training, and all queries are answered directly by their fine-tuned models without retrieval. The only difference is the selection of PEFT training data: our method selects semantically concentrated core samples, while the random baseline selects an equal number of samples randomly. We evaluate both methods on the same users and test queries from Table 3, with the results reported below.

Personalized Movie Tagging (Acc.↑)

MethodUser1 (Subsets Ratio:40%)User2 (Subsets Ratio:60%)User3 (Subsets Ratio:80%)
Random-subset PEFT48.350.065.3
Core-subset PEFT (Ours)52.954.367.3

Personalized Scholarly Title Generation (R-1↑)

MethodUser1 (Subsets Ratio:40%)User2 (Subsets Ratio:60%)User3 (Subsets Ratio:80%)
Random-subset PEFT44.146.748.3
Core-subset PEFT (Ours)47.349.649.1

The results show that semantic concentration-based data selection consistently outperforms equal-size random subsets across both tasks and all six user settings, demonstrating the effectiveness of selecting semantically concentrated core samples for PEFT. We will add these results to the revised version.

In addition, regarding fixed PEFT allocation ratios, Table 1 already reports the performance of different PEFT allocation ratios across user groups with varying semantic concentration levels. The results show that no single fixed ratio consistently achieves the best performance across all user groups, supporting the benefit of adapting allocation ratios based on semantic concentration.

[W1.2] The analysis in Figure 5 supports the concentration score's relationship with the selected ratio, but does not directly show how much the long-tail compensation term improves generation quality.

Response: To further evaluate the contribution of the long-tail compensation term to generation quality, we conduct an additional ablation study beyond the analysis in Figures 5 and 6. Specifically, we select users with similar average concentration scores (S_avg between 0.70 and 0.74) and divide them into two groups based on the strength of the long-tail compensation term: users with stronger long-tail characteristics (π_tail h_tail > 0.01) and users with weaker long-tail characteristics (π_tail h_tail ≤ 0.01). We then compare the generation performance of PUR with and without the compensation term (λ = 1 vs. λ = 0) on the same users, keeping all other settings unchanged. The results are reported below.

Personalized Movie Tagging (Acc.↑)

User GroupWith Compensation (λ=1)Without Compensation (λ=0)
Strong-tail Users65.062.9
Weak-tail Users64.864.8

Personalized Scholarly Title Generation (R-1↑)

User GroupWith Compensation (λ=1)Without Compensation (λ=0)
Strong-tail Users50.649.3
Weak-tail Users50.950.9

The results demonstrate that the long-tail compensation term improves task performance for Strong-tail Users by helping select more appropriate PEFT allocation ratios, while it has no impact on the performance of Weak-tail Users.

AuthorsOpenReview ↗

Author's Response (2)

[W2&Q1] How reliable is the inference routing decision? For queries assigned to the PEFT region, how often would retrieval have produced a better response?

Response: To answer this question, we conduct an additional experiment on the PEFT-Region Queries (PRQs) under the same experimental settings as in Table 3. Specifically, using the same fine-tuned PUR model, we compare two inference settings for queries assigned to the PEFT region: (i) directly generating responses without retrieval and (ii) generating responses with additional RAG retrieval. We also calculate the proportion of queries where retrieval produces a better prediction (With RAG > Direct Ratio). The results for the Personalized Movie Tagging task are reported below.

UserDirect Acc. ↑With RAG Acc. ↑With RAG > Direct Ratio
User1 (Sconc:0.689)70.073.33.3%
User2 (Sconc:0.752)67.369.23.8%
User3 (Sconc:0.811)70.773.14.9%

The results show that, for queries assigned to the PEFT region, only 3.3%–4.9% of queries benefit from additional retrieval. These findings support the effectiveness of our inference routing strategy in identifying queries that can be directly handled without retrieval, thereby reducing inference latency while maintaining competitive generation quality.

[W3] The writing feels somewhat amateurish and lacks the level of academic rigor expected for a conference paper.

Response: Thank you for your feedback. (i) We agree that some expressions, such as "the best performed compared method," are grammatically awkward, and the overall statement may also be confusing. We will revise the sentence "Extensive experiments across multiple tasks show that PUR consistently outperforms other methods for users with different semantic concentration, while reducing inference latency by 14.4%–46.6% compared with the best performed compared method." to "Extensive experiments across multiple tasks show that PUR consistently outperforms other methods for users with different semantic concentration levels, reducing inference latency by up to 46.6% compared with SOTA methods." (ii) We respect that academic writing styles and preferences may vary. However, we would like to clarify that some of the expressions highlighted by the reviewer were intentional writing choices rather than oversights. For example, the quotation marks around "core" and "peripheral" were used to distinguish different semantic regions. Similar terminology has also been used in recent conference papers [1-2]. (iii) "Fast and accurate" is a nickname we gave to our approach, intended to summarize the two main performance improvements of our work and make the presentation more engaging without compromising technical rigor. We fully respect different stylistic preferences, and we also believe that these writing choices do not affect the correctness of our proposed method or the validity of our experimental results.

[1] Han Lu et al. Boundary Matters: A Bi-Level Active Finetuning Method. NeurIPS, 2024.

[2] Chandra Sekhar Mukherjee and Jiapeng Zhang. Balanced Ranking with Relative Centrality: A Multi-Core Periphery Perspective. ICLR, 2025.

[Q2] In Appendix A.2, the authors reuse allocation ratios obtained with BGE when evaluating E5 and GTE, even though the semantic concentration scores change. How would PUR perform if the allocation ratios were determined directly from the new embedding models? Would the method require recalibration for each embedding model?

Response: Yes, different embedding models may produce different similarity scales, so the numerical mapping from semantic concentration scores to allocation ratios may require model-specific recalibration. However, this only introduces a one-time offline calibration cost for each embedding model. The core contribution of our work is to demonstrate the benefits of allocating workloads between PEFT and RAG in improving task performance and reducing online inference latency for users with different historical data distributions. Such recalibration only concerns the score-to-ratio mapping and does not change PUR's advantages in reducing online inference latency and improving task performance.

I appreciate the responses from the authors, I will raise my score accordingly.

Thank you very much. We sincerely appreciate your time and effort.

Reviewer_eydQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PEFT Union RAG (PUR), which uses the semantic concentration of user histories to divide data between personalized fine-tuning and retrieval. Concentrated core data are used for PEFT, while peripheral data remain available for RAG. At inference time, queries close to a core region bypass retrieval. Experiments on LaMP evaluate generation quality, inference latency, and the transfer of the allocation rule across tasks.

Critical strengths
  1. The paper considers both personalization quality and inference latency, addressing an important trade-off in personalized LLM services.
  2. The analysis suggests that the relative performance of PEFT and RAG depends on the semantic concentration of user histories.
  3. The paper tests the allocation rule on tasks outside the initial probing study and provides additional analyses of embedding models and computational costs.
Critical weaknesses
  1. Table 2 derives allocation ratios from Table 1, but the paper does not clearly establish a separate validation set for selecting these ratios.
  2. Random subsets of equal size, fixed allocation ratios, and routing-only variants are needed to isolate the contribution of semantic concentration.
  3. Comparisons against adaptive retrieval and tuned top-k settings are needed to establish PUR’s quality–latency advantage.
Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Author's Response

Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment on weaknesses and questions.

[W1] Table 2 derives allocation ratios from Table 1, but the paper does not clearly establish a separate validation set for selecting these ratios.

Response: We would like to clarify that Table 1 presents an exploratory study examining the effects of different PEFT allocation ratios across semantic concentration levels on two tasks. The purpose is to empirically examine the effectiveness of different allocation settings and derive the allocation rule summarized in Table 2 based on the observed performance. The generalizability of this allocation rule is further evaluated on four additional tasks, as shown in Figure 3.

[W2] Random subsets of equal size, fixed allocation ratios, and routing-only variants are needed to isolate the contribution of semantic concentration.

Response: To isolate the contribution of semantic concentration-based data selection, we conduct a controlled comparison with equal-size random subsets. Specifically, to exclude the influence of inference routing, both methods use the same amount of data for PEFT training, and all queries are answered directly by their fine-tuned models without retrieval. The only difference is the selection of PEFT training data: our method selects semantically concentrated core samples, while the random baseline selects an equal number of samples randomly. We evaluate both methods on the same users and test queries from Table 3, with the results reported below.

Personalized Movie Tagging (Acc.↑)

MethodUser1 (Subsets Ratio:40%)User2 (Subsets Ratio:60%)User3 (Subsets Ratio:80%)
Random-subset PEFT48.350.065.3
Core-subset PEFT (Ours)52.954.367.3

Personalized Scholarly Title Generation (R-1↑)

MethodUser1 (Subsets Ratio:40%)User2 (Subsets Ratio:60%)User3 (Subsets Ratio:80%)
Random-subset PEFT44.146.748.3
Core-subset PEFT (Ours)47.349.649.1

The results show that semantic concentration-based data selection consistently outperforms equal-size random subsets across both tasks and all six user settings, demonstrating the effectiveness of selecting semantically concentrated core samples for PEFT. We will add these results to the revised version.

In addition, regarding fixed PEFT allocation ratios, Table 1 already reports the performance of different PEFT allocation ratios across user groups with varying semantic concentration levels. The results show that no single fixed ratio consistently achieves the best performance across all user groups, supporting the benefit of adapting allocation ratios based on semantic concentration.

[W3] Comparisons against adaptive retrieval and tuned top-k settings are needed to establish PUR’s quality–latency advantage.

Response: Thank you for this suggestion. We conduct additional comparisons under different retrieval settings. Specifically, we evaluate RAG, Full PEFT+RAG, and PUR with different Top-k values (k=2 and 4). We use the same experimental settings as in Table 3. The results for the Personalized Scholarly Title Generation task are reported below.

Top-kMethodUser1 (Sconc:0.701) Overall Perf. (R-1) ↑User1 (Sconc:0.701) Avg. Infer. Time(s)↓User2 (Sconc:0.743) Overall Perf. (R-1) ↑User2 (Sconc:0.743) Avg. Infer. Time(s)↓User3 (Sconc:0.809) Overall Perf. (R-1) ↑User3 (Sconc:0.809) Avg. Infer. Time(s)↓
k=2RAG48.21.8748.61.8149.01.90
k=2Full PEFT+RAG46.81.9350.31.9150.12.08
k=2PUR (Ours)49.31.6151.41.4851.31.40
k=4RAG48.12.5749.02.6449.42.51
k=4Full PEFT+RAG47.12.6450.22.7050.32.66
k=4PUR (Ours)49.22.2051.61.9351.61.53

The results show that PUR consistently achieves the best task performance across all three users under both Top-k settings, while reducing average inference latency compared with RAG and Full PEFT+RAG. These results further demonstrate the effectiveness of PUR's workload allocation strategy in improving generation quality and reducing average inference latency under different retrieval settings.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on Author Response

Dear Reviewer,

Thank you for your valuable and constructive comments. Could you please let us know whether our responses have addressed your concerns? We would greatly appreciate your feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.