The main concern of this reviewer is the novelty and significance of the contributions. While the experimental analysis is carefully conducted and well presented, most of the main observations have already been investigated in previous literature or are relatively straightforward.
In particular, combining original documents with question-answer pairs and alternative formulations has been extensively investigated in previous works [1–4]. The proposed MIX3 is a straightforward combination of existing approaches, and the observation that more diverse supervised text improves knowledge incorporation is already well established.
Similarly, applying LoRA to all linear projections and the performance advantages of adapting additional modules have been investigated in previous works [5, 6]. While the interaction between LoRA placement and training objectives is an interesting empirical observation, the experiments do not separate the effect of placement from the increased number of trainable parameters.
Overall, this reviewer finds the empirical analysis useful and well presented, but the novelty and significance of the contributions insufficient for ICLR in the manuscript's current form.
Please also refer to the questions below.
[1] Zeyuan Allen-Zhu and Yuanzhi Li. "Physics of Language Models: Part 3.1, Knowledge Storage and Extraction." Proceedings of the 41st International Conference on Machine Learning (ICML), Proceedings of Machine Learning Research, vol. 235, pp. 1067–1077, 2024.
https://proceedings.mlr.press/v235/allen-zhu24a.html
[2] Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. "Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 237–250, 2024.
https://doi.org/10.18653/v1/2024.emnlp-main.15
[3] Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih, and Srinivasan Iyer. "Instruction-tuned Language Models are Better Knowledge Learners." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), vol. 1, pp. 5421–5434, 2024.
https://doi.org/10.18653/v1/2024.acl-long.296
[4] Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, and Furu Wei. "Instruction Pre-Training: Language Models are Supervised Multitask Learners." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2529–2550, 2024.
https://doi.org/10.18653/v1/2024.emnlp-main.148
[5] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. "QLoRA: Efficient Finetuning of Quantized LLMs." Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023.
https://proceedings.neurips.cc/paper_files/paper/2023/hash/1feb87871436031bdc0f2beaa62a049b-Abstract-Conference.html
[6] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. "LoRA Learns Less and Forgets Less." Transactions on Machine Learning Research (TMLR), 2024.
https://openreview.net/forum?id=aloEru2qCG
Thank you for your review and thoughtful comments!
Table. Closed-book accuracy on SQuAD, Qwen2.5-7B. † lr 1e-4, since lr 1e-3 collapses.
With parameters fixed, changing placement raises MIX3 by 12.6 ±1.8 and leaves SYN unchanged. With placement fixed, 8× more parameters raise MIX3 by 9.4 ±2.0 and lower SYN by 6.5 ±2.0. So whether capacity helps depends on the objective. The same kind of control runs through the paper. Removing passage tokens from the loss of SYN lowers the gain steadily from 14.9 to −3.7, whereas removing question tokens from the pairs has no significant effect (Figure 1a, Appendix D.1).
Changes in the paper. We describe MIX3 as a reference objective in the abstract and Section 1, add the placement control to Section 5.2, and report factual and counterfactual judgment separately in Table 8.