acceptodds
Under review as a conference paper at ICLR 2027

Rethinking What LLMs Transfer to Recommendation: Localizing and Recombining Pretrained Computation

Abstract

Large language model (LLM) initialization improves recommendation, with its benefits often attributed to general capabilities that existing methods seek to preserve. However, recovering general-task performance through mixed supervision brings no clear additional recommendation gain. Yet shared-neuron ablations reveal continued dependence on resources used by general tasks. What, then, does pretrained initialization supply that recommendation actually uses? We derive a cross-entropy training-gain bound in a simplified feed-forward model, separating the improvement available from initial computation from the cost of learning its recombination. Neuron-level analyses support this account: many channels used by the trained recommender are identifiable before training and remain useful as adaptation changes their task allocation. These findings motivate initial localization-guided neuron recombination, which selects channels from their initial activity on recommendation contexts and learns a low-rank residual mapping to recombine their responses after recommendation fine-tuning. Across three domains and four LLM backbones, our method consistently improves over standard fine-tuning, and ablations supporting both initial localization and learned recombination.

Primary area:
generative models
Keywords:
Recommender Systems, Large Language Models, Post-Training, Transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HxVrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates what LLM pretrained initialization contributes to recommendation. It argues that recovering general-task performance does not bring clear additional recommendation gains, while recommendation still depends on task-shared neurons. Based on this, the paper proposes an initial localization-guided neuron recombination method: it first identifies candidate channels based on FFN channel activations on recommendation contexts at initialization, then performs recommendation SFT, and finally freezes the SFT backbone while training only a low-rank residual recombination module applied to the selected channels.

Critical strengths

1,The paper is generally well written, and the method, experimental setup, and main findings are relatively easy to understand.

2,The experiments are relatively extensive. The method achieves consistent, though limited in magnitude, improvements over standard recommendation SFT across multiple metrics.

Critical weaknesses

1,Section 3.2 presents a two-factor FFN with fixed directions, linear readout, and gradient-flow analysis, leading to Propositions 1 and 2. However, the actual module is a low-rank residual applied to a frozen SFT backbone, rather than the two-factor (uj,vj)(u_j,v_j) update assumed by the theoretical model.

2,The ablation studies are insufficient: the method’s gains may also stem from additional low-rank parameter capacity, the second-stage single-target SID supervision.

Questions for authors

How large is the drift of the initially localized channels after SFT? Is re-localization after SFT equally effective?

Reviewer_QeH6OpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the contribution of pretrained initialization in large language models (LLMs) to downstream recommendation tasks. The authors observe that although pretrained initialization improves recommendation performance, recovering general-task capabilities through mixed supervision does not necessarily yield additional recommendation gains. To explain this observation, the paper presents a theoretical analysis that separates the benefits of initial computation from the cost of learning its task-specific recombination. Based on neuron-level localization and intervention analyses, the authors propose an initial localization-guided neuron recombination method that identifies useful feed-forward network (FFN) channels before adaptation and learns a low-rank residual mapping to recombine their activations after recommendation fine-tuning. Experiments across three recommendation domains and four LLM backbones demonstrate consistent improvements over standard supervised fine-tuning (SFT).

Critical strengths
  1. The paper investigates an interesting and relevant research question concerning the relationship between general-purpose LLM capabilities and downstream recommendation performance. The finding that recovering general-task performance does not necessarily improve recommendation, while initially shared neurons remain important, provides an interesting perspective on knowledge transfer.

  2. The neuron localization and intervention analyses provide empirical evidence that pretrained computation remains useful after recommendation fine-tuning. These analyses help motivate the proposed approach and provide insights into the reuse of pretrained model components.

  3. The theoretical derivations and extensive experiments provide useful insights into the motivation behind the proposed method and its effectiveness.

Critical weaknesses
  1. The technical novelty is limited. The proposed method can be interpreted as applying low-rank adaptation to a selected subset of FFN channels, which has been extensively investigated in the importance-based parameter selection and low-rank adaptation.

  2. The literature review in this paper is not sufficient. It would be better to add a discussion of related work on advanced importance-aware or sparse low-rank adaptation methods, such as AdaLoRA [1] and RoseLoRA [2]. [1] Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W. and Zhao, T., 2023. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. The Eleventh International Conference on Learning Representations, 2023. [2] Wang, H., Liu, T., Li, R., Cheng, M.X., Zhao, T. and Gao, J., 2024, November. Roselora: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 996-1008).

  3. Although the authors compare their method against standard SFT, unrestricted LoRA, and random channel selection, comparisons with more advanced importance-aware or sparse low-rank adaptation methods are missing.

  4. The experimental validation of the localization strategy is limited. Additional experiments comparing alternative importance criteria, different channel-selection thresholds, and different numbers and distributions of selected channels could strengthen the paper's contributions.

  5. All three recommendation datasets come from the same benchmark framework, limiting the evaluation of the proposed method's generalizability.

Questions for authors

Beyond improving overall recommendation accuracy, have the authors observed whether localized neuron recombination affects particular types of recommendations, such as those involving sparse interaction histories, semantically related items, or less frequently interacted items?

Reviewer_jxFXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what pretrained LLM initialization contributes to recommendation beyond observable general-task performance. Under the studied training recipes, mixed supervision restores general-task scores without a clear additional recommendation gain, while interventions show that some initially shared neuronal positions remain functionally important after recommendation fine-tuning. The authors derive a cross-entropy training-gain bound in a simplified feed-forward model, separating the improvement available from initial computation from the cost of learning its recombination. They then propose initial localization-guided neuron recombination: select FFN channels from their activity on recommendation prompts before training, perform ordinary recommendation SFT, and learn a supported low-rank residual with the backbone frozen. Experiments cover three domains and four backbones, with consistent point-estimate improvements over the corresponding SFT models. Channel-selection, residual-strength, and assisted-output experiments provide additional evidence for the design and its motivating observations.

Critical strengths
  1. The paper provides complementary evidence for a useful distinction between general-task performance and the resources available for recommendation. Behavioral comparisons, position-level analyses, functional interventions, and assisted-output evaluations jointly support the view that these quantities need not move together. In Appendix G, removing an initially shared subset reduces recommendation MRR from 7.2601 to 1.8297, whereas matched random removal gives 7.2482. This is stronger evidence of continued functional involvement than overlap statistics alone. The theoretical analysis is also internally coherent under its stated assumptions; I did not identify an error in the two propositions.

  2. The proposed adaptation design has empirical support beyond a single model or domain. Table 2 reports improvements in all four metrics across twelve backbone-domain configurations, with relative R@32 gains of approximately 1.82%–5.68%. Importantly, Section 5.3 compares initial localization with per-layer count-matched random selection and unrestricted down-projection LoRA from the same SFT checkpoint, using the same supervision and training budget in the Qwen-1.7B/Ad setting. This control provides substantive evidence for the value of the selected input support, rather than attributing all gains to the additional training stage or the change to single-SID targets.

Critical weaknesses
  1. The reporting of the main method experiments is insufficient to fully assess reproducibility and the reliability of the relatively small gains (Sections 5.2–5.3, Table 2, and Appendix D). Table 2 states p<0.05p < 0.05 without clearly specifying the test, comparison, unit of inference, or source of uncertainty. The fixed-checkpoint bootstrap described for the analysis experiments does not clearly establish the statistical procedure for the main method results. The second-stage learning rate and training duration, as well as the rank, inference-time residual scale, and checkpoint-selection rules for the main configurations, are also not fully reported. The paper should provide these settings and uncertainty estimates for the key controlled comparison. These are reporting gaps; they do not establish that repeated runs or appropriate tests were not performed.

  2. The evidence more directly supports continued usefulness of initially localized positions than preservation and recombination of their original computational content (Section 3.3 and Appendices B.4, C.6, and F). Full-parameter SFT can change the responses and downstream roles of these positions. The intervention controls match layer, projection, and count, but not local contribution magnitude, so they do not isolate a special causal role for task-sharing itself. Moreover, the fixed-feature coefficient-residual theory is not a guarantee for the implemented two-stage low-rank method. The authors acknowledge most of these boundaries in the appendices, so I regard this primarily as a limit on the strength of the mechanistic contribution, rather than a demonstrated error. The main-text claims should consistently distinguish functional reuse of positions from preservation of computational content; stronger claims would require additional discriminating evidence.

Questions for authors
  1. Statistical evidence for the main results: What exact statistical procedure supports the p<0.05p < 0.05 statement in Table 2? Please specify the comparisons, unit of inference, test, and whether the uncertainty estimates account for training randomness. Please also report uncertainty for the central controlled comparison in Figure 3a. Since the reported gains are relatively modest, this information is important for assessing their reliability. If the additional analyses provide sufficiently robust evidence, I would be open to reconsidering my score.

  2. Implementation clarification: Please report the second-stage learning rate, training duration, rank, inference-time beta, and checkpoint-selection rules used for the main results. Does the selected support include all channels identified from recommendation prompts, or is it further intersected with the general-task set? The wording in Section 4.1 can suggest the latter, while Appendix D.1 appears to define the former. This distinction matters for reproducing the method.

  3. Claim clarification: Is the claim of pretrained computational reuse intended to mean that initially localized positions remain functionally important, or that their original computational content is preserved? The current evidence is more direct for the former. Please identify any evidence that specifically distinguishes the latter, or align the main-text interpretation with the more limited claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.