- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a fast and accurate method to improve personalized LLM performance, named PEFT Union RAG (PUR). It splits workload between PEFT and RAG according to their semantic concentration, aiming to enhance the performance of personalized LLMs with better generation accuracy/quality and lower inference latency. Records close to a semantic center are used for fine-tuning, while the remaining records form the retrieval corpus. During the online inference phase, if a user's query falls within the core semantic region covered by PEFT, the model generates a response directly; otherwise, RAG is invoked to retrieve external knowledge.
Critical strengths
- The paper addresses a practically important problem. Personalized models must exploit user histories to improve generation quality while maintaining acceptable inference latency, making the joint consideration of quality and latency highly valuable.
- PUR uses the distribution of user histories to determine both the fine-tuning subset and the retrieval path, providing an intuitive approach to coordinating coordinating parametric fine-tuning and retrieval augmentation.
- Experimental results show consistent improvements of PUR on better generation accuracy and shorter inference latency for various users across multiple tasks.
Critical weaknesses
- PUR maps the semantic concentration metric to the PEFT data allocation ratio through an empirically calibrated look-up table (Table 2), rather than learning the allocation adaptively. Its applicability to new domains or users with substantially different historical data distributions requires additional justifications.
- Users with very limited historical data may not provide enough samples to estimate a reliable semantic center, and such unstable estimates could produce unreliable inference-path selection or lead to an excessive reliance on RAG. Have the authors considered the robustness of the PUR framework under these low-data conditions?
- Recent research has already explored personalized LLMs utilizing a full combination approach (100% PEFT + 100% RAG). The paper should more clearly distinguish the contribution of PUR and compare against representative existing methods to justify the proposed mechanism.
- The experiments only use Llama-2-7B as the base LLM, leaving it unclear whether this semantic concentration-based allocation strategy remains effective when scaled to larger LLMs or other SOTA models with stronger in-context learning capabilities.
Questions for authors
Please refer to the weakness above
3 comments
I appreciate the authors' efforts. My major concerns have been addressed. I raise my score from 2 to 3.
Thank You and Rating Update
Dear Reviewer,
Thank you very much for your positive feedback and for reconsidering your assessment. We truly appreciate your time and support.
We noticed that the rating in the Official Review still shows 2. If possible, could you kindly update it to reflect your revised score of 3?
Thank you again for your consideration.
Best regards,
The Authors
Author's Response
Thank you for your valuable comments! Please find below the responses to each comment on weaknesses.
Response: We agree that new domains with substantially different historical data distributions may require recalibration of the mapping between semantic concentration scores and allocation ratios. However, this only introduces a one-time offline calibration cost for new domains. The core contribution of our work is to demonstrate the benefits of allocating workloads between PEFT and RAG in improving task performance and reducing online inference latency for users with different historical data distributions. Such recalibration only concerns the score-to-ratio mapping and does not change PUR's advantages in reducing online inference latency and improving task performance.
Response: Thank you for raising this concern. We agree that PUR may be less reliable under extremely limited historical data conditions. However, such extreme data scarcity also poses challenges for standard PEFT and RAG methods. We have already investigated the impact of historical data size on PUR's performance in Figure 4. The results show that PUR consistently outperforms the compared methods across different semantic concentration levels and historical data sizes, even in settings with as few as 40–50 historical records. These results provide empirical evidence that PUR can maintain its performance advantage under relatively limited historical data conditions.
Response: We would like to clarify that the core contribution of PUR is a novel semantic concentration-based workload allocation strategy between PEFT and RAG (e.g., 60% PEFT and 40% RAG according to semantic concentration). We never claimed that we proposed the full combination (100% PEFT + 100% RAG) as a new approach. Actually, we discussed SOTA full combination methods [1-2] in Section 2. Moreover, we have explicitly included Full PEFT+RAG as a compared method in Section 5.
[1] Z. Tan et al. Democratizing Large Language Models via Personalized Parameter-Efficient Fine-Tuning. EMNLP, 2024.
[2] B. Pingua et al. Medical LLMs: Fine-Tuning vs. Retrieval-Augmented Generation. Bioengineering, 2025.
Response: Thank you for this comment. We conduct additional experiments using Llama-2-13B and Llama-3.1-8B as base models, following the same experimental settings as Table 3. The results on the Personalized Scholarly Title Generation task are reported below.
The results show that PUR consistently achieves the best task performance across all three users under both Llama-2-13B and Llama-3.1-8B, while reducing average inference latency compared with the retrieval-based methods. These results demonstrate the effectiveness of our semantic concentration-based workload allocation strategy across different LLMs.