acceptodds
Under review as a conference paper at ICLR 2027

LoopFM: Learning frOm HistOrical RePresentations of Foundation Model for Recommendation

Abstract

Knowledge distillation (KD) transfers a single scalar prediction from a large foundation model (FM) to compact vertical models (VMs), suffering from diminishing transfer ratio—the fraction of FM improvement captured by the VM—as a single scalar cannot convey the rich intermediate knowledge that larger FMs learn. To address this bottleneck, we propose **LoopFM** (**L**earning fr**O**m Hist**O**rical Re**P**resentations of **FM**), a framework that opens a high-bandwidth transfer channel by structuring FM intermediate embeddings as *input features* (e.g., user history sequence) for downstream VMs, without requiring real-time FM inference at serving and architectural coupling between FM and VM. We provide a theoretical framework for LoopFM with a gain decomposition and transfer-ratio analysis. On three public benchmarks, LoopFM demonstrates strong relative AUC improvements (e.g., 6%+ on TaobaoAd) and complementary knowledge transfer capability with KD. On industrial-scale systems (billions of examples, trillion-parameter FMs), LoopFM approximately **doubles** the knowledge transfer ratio on top of KD, delivering a +0.5% conversion improvement in the first half after its initial launch, and a +1.03% and +1.22% conversion improvement from two individual launches in the subsequent half. Through systematic experiments, LoopFM demonstrates a predictable scaling law in sequence length, embedding dimension, and upstream FM size.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Recommender Systems, Knowledge Transfer, Knowledge Distillation, Foundation Models, Supervised Representation Learning, User Behavior Sequence, Scaling Laws

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Yb6LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

LoopFM transfers intermediate information from a recommendation FM to compact serving models alongside scalar KD. It compresses offline FM activations with an autoencoder and organizes past-interaction embeddings into temporally ordered user-history features, avoiding FM inference during serving. The population analysis separates temporal-history information, additional FM-derived information and a pipeline residual, alongside a conditional transfer-ratio analysis. Public experiments show AUC gains over KD on TaobaoAd, with more mixed results elsewhere, and industrial deployments report conversion gains. Comparisons with simpler history and embedding-transfer alternatives assess the pipeline's incremental value, while representation, compression and encoder studies examine practical design choices.

Critical strengths
  1. The staged pipeline description makes the offline extraction, compression and serving interface easy to follow. The experimental questions connect the design choices to specific comparisons.

  2. Offline storage of past-interaction embeddings provides a practical interface for compact serving models. The compression and encoder studies address the storage and capacity constraints of that design.

  3. The population information decomposition usefully distinguishes raw historical signal from additional FM-derived information and residual history information. This gives the history-based comparisons a clear analytical context.

  4. TaobaoAd AUC gains across student architectures and a repeated-seed check for DeepFM support the method's empirical value. Reported production conversion gains add evidence of deployment utility.

  5. Strong pooling, raw-history and item-only baselines help assess the incremental value of the full pipeline. Layer and checkpoint comparisons also reveal practical tradeoffs in choosing representations.

Critical weaknesses
  1. A1 in Sec. 4.2 gives Eq. (6) that presented as a consequence of NTK and benign overfitting, but Appendix F.7 does not establish the required BCE excess-risk envelope: classification-decision equivalence does not control BCE risk, and the stated benign limits do not establish the relative rate needed to absorb the bias term. Treating the BCE envelope as an independent assumption would preserve the conditional analysis; the claimed derivation needs a loss-specific proof or narrower wording.

  2. Eq. (8) in Theorem 2(2) omits the nonnegativity condition on the numerator's lower bound stated in Appendix F.4. Assumption A3 does not imply this condition, which is needed for the proof's denominator substitution. Please state it in the main theorem.

Questions for authors
  1. What fitted relation and predictive check support the power-law claim in Fig. 2, and which NE reference and fixed teacher, data, compression and student settings define the comparison? Otherwise, please describe the evidence as an observed trend.
2 comments

We thank the reviewer for the careful reading of the theory.

Q1. A1 / Eq. (6) is presented as derived, but Appendix F.7 does not establish the BCE excess-risk envelope.

We agree. The NTK and benign-overfitting results we cite give classification-decision equivalence and benign limits for the linearized model. They do not give a two-sided BCE excess-risk envelope at the relative rate needed to absorb the bias term.

The revision states the two-sided envelope in Eq. (6) as an explicit assumption (A1), in Sec. 4.2 and Appendix E. Appendix F.7 is retitled as motivation under the stated NTK and benign-overfitting conditions, rather than a derivation, and its cross-entropy paragraph no longer claims that Bartlett's bounds transfer to the BCE excess risk. The corollary in F.7 that recovers the envelope is now stated for the linearized squared-loss surrogate only, and the general-covariance bound (Appendix F.8) is stated with the assumed envelope. Appendix F.7's corollary now states that the linearized (NTK / benign-overfitting) result gives the two-sided envelope only for the linearized squared-loss surrogate; for the BCE loss the envelope is an assumption (A1), not a derived result. As the reviewer notes, all downstream results (Theorem 2, Corollary 4) are conditional on this envelope and remain valid as conditional statements; the revision says so in Sec. 4.2 and in the Limitations.

W2. Eq. (8) in Theorem 2(2) omits the nonnegativity condition from Appendix F.4.

The proof substitutes the denominator bound, which requires the numerator's lower bound to be nonnegative. A3 does not imply this, and it is necessary: if the numerator's lower bound is negative, dividing it by an upper bound on the denominator no longer gives a valid lower bound. The revision adds it to the statement of Theorem 2(2) (as condition (⋆), so the existing equation numbers are unchanged), to Corollary 4 and to the general-covariance version of the bound (Appendix F.8), and notes that it holds whenever the feature-gap term dominates the temporal pipeline loss, i.e. δ ≥ δ_min = τ₂ I_temporal / ((1 − η₁) κ^hist_gap).

Q. Fig. 2: what fitted relation and predictive check support the power-law claim? Which NE reference and fixed settings define the comparison?

Fit on Fig. 2's four plotted points (compute 0.84 / 2.00 / 2.90 / 5.00 × 10²³ FLOPs → NE gain 0.13 / 0.24 / 0.37 / 0.57%):

FitFormulaR²Prediction for the largest point from the other three (observed 0.57%)
Power law0.146 · C^0.840.992 (log–log)0.549%
Log-linear (the dashed line currently drawn)0.134 + 0.559 · log₁₀ C0.9260.443%

The power law fits better and predicts the held-out point within 0.02 percentage points. In the revision we:

  • redraw Fig. 2 with the power-law fit (the current dashed line is the log-linear fit);
  • report the fit and the held-out check;
  • describe the result as "consistent with a power law (4 points)" rather than a predictive law.

Settings. Each point in Fig. 2 is taken from one LoopFM launch on the same production VM model, distilled from a different version of the FM; we report each FM version's training compute (FLOPs) and LoopFM's relative NE gain. Unfortunately, we do not have enough time within the rebuttal period to obtain legal approval to release the detailed setup of these internal results. We hope the fitted relation and the held-out check address the core of the question, and the revision softens the wording as described.

Thank you for the clarifications and revisions, which address my main concerns. I will therefore raise my Soundness score from 2 to 3 while retaining my weak acceptance. I have no further questions.

Reviewer_XQ9BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoopFM, a new framework for transferring knowledge from large recommendation foundation models (FMs) to lightweight vertical models (VMs). Rather than relying solely on scalar knowledge distillation, LoopFM extracts intermediate FM representations, compresses them, and organizes historical interaction embeddings into user sequences that downstream VMs can consume without real-time FM inference. The paper also adds theoretical analysis and extensive experiments to show the merits of the propsed method.

Critical strengths
  1. The research question is both practical and meaningful, transferring rec knowledge from large models to vms is needed.

  2. Practical validation on the large-scale recommendation datasets. The experimental analysis also provides meaningful insights, including fm layer selection, embedding compression, sequence length, and downstream sequence encoders. The comparison between raw historical sequences and FM-derived representations is informative. The observation that checkpoint updates may introduce representation drift is also relevant for practical deployment.

Critical weaknesses
  1. Limited method novelty and insufficient closely related works. Compressing historical interaction representation and using it as downstream sequence features is common in previous works, such as [1]. These approaches construct compact instance-level representations from upstream models and organize them into historical sequences for downstream recommendation.

  2. I also has concerns whether the performance gains come from improved knowledge transfer. In table4, raw User Seq achieves an AUC of 0.6302, compared with 0.6319 for LoopFM.

  3. I also have concerns about the word 'FM'. In public experiments, they primarily use relatively small FMs, leaving the proposed benefits of transferring information across a substantial FM–VM feature gap untested.

  4. The improvements are less consistent outside TaobaoAd dataset. Please give the analysis and reason

[1] IAT: Instance-As-Token Compression for Historical User Sequence Modeling in Industrial Recommender Systems

Questions for authors

see weakness

2 comments

Q1. Novelty vs IAT [1] and related works. We thank the reviewer for raising this point. See "General response to all reviewers" above.

Q2. Do the gains come from knowledge transfer? See response to reviewer xgN4 (https://openreview.net/forum?id=hrYycfhjgH&noteId=UBtwqhAXkn) To add on that, in internal production, LoopFM has been iterated together with the FM in multiple internal launches, each replacing the previous version; the online gain grows with the FM generation (Sec. 5.4, Fig. 2). If the gain came only from having a sequence, upgrading the FM would not change it. In public benchmark, we reproduce this setting by comparing LoopFM from trained and untrained FM of the same architecture and features, keeping the sequence construction unchanged. The result is summarized in Table 4 below and the full experiment setting and result can be found in Table 24 (Appendix L.4) of the revised paper.

Table 4: LoopFM's gain with an untrained vs. trained FM (FM: DMIN, 3 seeds).

SettingFMVMBaselineLoopFM, untrained FM (share of the gain)LoopFM, trained FM
TaobaoAd, FM and VM see the same features (Table 4 setting)DMINDMIN0.59370.6307 (95%)0.6328
TaobaoAd, the VM lacks features the FM seesDMINDeepFM0.58250.6245 (88%)0.6302
KuaiVideo, FM and VM see the same featuresDMINDMIN0.71680.7190 (33%, n.s.)0.7234
Amazon, item-keyedDMINDIEN0.86150.8638 (31%)0.8689
Amazon, item-keyedDMINDMIN0.85640.8580 (20%)0.8642
Amazon, item-keyedDMINDMR0.86080.8614 (13%)0.8656
Amazon, item-keyedDMINDeepFM0.84120.8428 (23%)0.8481

On TaobaoAd, where the FM and VM see the same features (Table 4 of Sec 5.3.4), the untrained FM already recovers 95% of the gain. With no feature gap, there is little cross-feature knowledge to transfer, so most of the gain comes from giving the VM a compact history; this explains why LoopFM is close to the full raw sequences in Table 4. Once the VM lacks features the FM sees, the FM's knowledge becomes visible. On TaobaoAd, where the VM lacks the 8 user-profile features and all the behaviour sequences, the trained FM then adds +0.0057 over the untrained one. On KuaiVideo, even with the same features, an untrained FM recovers only 33% of the gain (n.s.), so the trained FM matters there too. On Amazon, which also has no feature gap, the FM's training still matters: item-keyed sequences from an untrained FM give only +0.0006 to +0.0023 AUC, against +0.0046 to +0.0078 from a fully trained FM.

Q3. "FM" in the public experiments is small; the feature-gap benefit is untested.

Feature gap (see details in Appendix. L.1 and L.3). We now test the feature-gap benefit directly, by removing features from the VM only while the FM keeps them. On all three datasets, LoopFM's sequence carries the FM's knowledge of the missing features to the VM, and its gain is significantly larger than that of KD (Table 5).

Table 5. Feature gap: features removed from the VM only (the FM keeps them); test AUC differences.

DatasetΔ test AUCFMFmFMDeepFMDIENDMRDMIN
TaobaoAdKD − BL+.0150+.0130+.0099+.0050+.0034+.0065
LoopFM − BL+.0551+.0509+.0467+.0477+.0463+.0488
KD+LoopFM − KD+.0457+.0403+.0424+.0431+.0427+.0434
KuaiVideoKD − BL———+.0041+.0071+.0088
LoopFM − BL———+.0124+.0137+.0107
KD+LoopFM − KD———+.0088 (n.s.)+.0061+.0042
AmazonLoopFM − BL——+.0429†+.0171+.0171+.0152

FM size and FM quality (Appendix G and L.4). Our claim presumes that LoopFM's gain scales with FM quality. A larger FM is a better FM only when it has enough rich training data. For example as in industry-scale RecSys, FMs have trillions of parameters, are trained on billions of examples per day, and each generation is trained with more compute (Sec. 5.4). Across four FM generations, LoopFM's relative NE gain grows from 0.13% to 0.57% as FM training compute grows from 0.84 to 5.0 × 10²³ FLOPs.

The public benchmarks cannot precisely reproduce this scaling, because four days of data are too few to train a larger FM well (Appendix G). On TaobaoAd, FMs with 3.9× and 13.7× the dense parameters, and a transformer FM, are all worse on the test day. Even so, the VM's LoopFM gain follows FM quality exactly:

Table 6: larger FMs may be worse, and the VM's LoopFM gain falls with FM quality. (details in Appendix L.4)

FMDense paramsFM test-day AUCVM + LoopFMΔ vs FM-S
FM-S (paper)1.1M0.57760.6297—
FM-M4.3M0.57590.6290−0.0007 (n.s.)
FM-L15.0M0.57340.6283−0.0014

On Amazon, wider FMs barely change FM quality, so we varied quality directly: the same FM untrained, trained for 1 epoch, on 25% or 50% of the users, or fully trained. The item-keyed LoopFM gain rises with FM quality on all four VMs, from +0.0006–0.0023 (untrained) to +0.0046–0.0078 (fully trained) AUC (Spearman ρ = 0.75–0.86). Together, the public and production results show that LoopFM transfers the knowledge of FM to the VM and the stronger FM leads to better transfer ratio. With enough data, as in production, larger FMs are better FMs, and LoopFM's gain grows with them.

Q4. Improvements are less consistent outside TaobaoAd. Analysis and reasons. We first reran the paper settings with 3 seeds (test AUC, mean ± sd over 3 seeds; bold = best method per VM). Results are presented in Table 7.

Table 7: Multi-seed reproduction (3 seeds; provided here, not added to the revised PDF). ‖ Amazon uses item-keyed LoopFM (Appendix L.3)

DatasetVMw/o distillKDLoopFM-onlyKD+LoopFM
TaobaoAdFM.5818±.0005.5937±.0003.6214±.0011.6306±.0005
FmFM.5845±.0002.5882±.0049.6197±.0019.6175±.0059
DeepFM.5884±.0000.5975±.0001.6232±.0005.6340±.0001
DIEN.5931±.0003.5966±.0001.6339±.0006.6339±.0003
DMR.5951±.0005.5993±.0002.6352±.0001.6338±.0027
DMIN.5941±.0008.6002±.0002.6326±.0004.6347±.0002
KuaiVideoDIEN.7102±.0014.7210±.0015.7195±.0025.7233±.0010
DMR.7185±.0008.7246±.0012.7232±.0017.7248±.0014
DMIN.7168±.0033.7229±.0027.7234±.0024.7229±.0012
Amazon‖DeepFM.8309±.0007.8359±.0011.8353±.0029.8384±.0013
DIEN.8512±.0012.8676±.0015.8561±.0041.8695±.0026
DMR.8506±.0022.8648±.0021.8536±.0045.8673±.0009
DMIN.8497±.0037.8656±.0015.8523±.0043.8668±.0028

With multiple seeds, the LoopFM gains are consistent across all three datasets. LoopFM significantly outperforms the no-distillation baseline on every VM of every dataset, and KD + LoopFM is the best method on 10 of the 13 VMs. In CTR prediction, AUC gains on the order of 0.001 are considered practically significant (Wang et al., 2021; Song et al., 2019).

References:

  • Wang et al. DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. WWW 2021.
  • Song et al. AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks. CIKM 2019.
Reviewer_xgN4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LoopFM, a framework for transferring knowledge from large foundation models (FMs) to compact vertical models (VMs) for recommendation. The method aims to address the diminishing transfer ratio of scalar knowledge distillation. LoopFM extracts and compresses FM intermediate representations, organizes historical embeddings into structured sequences, and uses them as additional VM input features. Experiments on three public datasets and industrial-scale systems show that LoopFM improves recommendation performance and approximately doubles the knowledge transfer ratio over KD in production.

Critical strengths
  1. LoopFM transfers historical FM representations to VMs without requiring real-time FM inference or architectural coupling between the models.
  2. The paper evaluates LoopFM on three public datasets and industrial-scale systems, showing performance improvements across different VM architectures and production settings.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines existing techniques, including intermediate representation extraction, autoencoder-based compression, and historical sequence modeling. In particular, IAT (Li et al., 2026) has already proposed a similar pipeline that compresses historical interaction representations and organizes them into sequences for recommendation. Although the authors introduce the transfer ratio as a different motivation and provide theoretical analysis, the theoretical results appear to be applicable to IAT under similar assumptions. Therefore, it remains unclear what key methodological novelty LoopFM introduces compared with IAT, and how its design specifically improves the transfer ratio.
  2. Limited improvement over raw historical sequences. Although LoopFM achieves significant improvements over the original VM, Table 4 shows that raw user behavior sequences already achieve 0.6302 AUC, while LoopFM achieves 0.6319 AUC under the same VM capacity. This small difference raises concerns about how much additional knowledge is transferred from the FM beyond historical sequence information. Moreover, this comparison is only conducted on TaobaoAd, making it difficult to determine whether LoopFM provides consistent advantages over raw historical sequences across different datasets.
Questions for authors
  1. Given that the theoretical analysis appears applicable to IAT as well, what specific methodological design distinguishes LoopFM from IAT in improving the transfer ratio?
  2. Could the authors extend the comparison with raw historical sequences in Table 4 to KuaiVideo and Amazon Electronics to verify whether LoopFM provides consistent improvements across different datasets?
3 comments
AuthorsOpenReview ↗

Response to Q1

Q1.1: Novelty vs IAT [1] and "how its design specifically improves the transfer ratio". We thank the reviewer for raising this point. See "General response to all reviewers" above.

To further demonstrate how LoopFM's design improves the transfer ratio, and quantify this advantage over IAT, we directly compared LoopFM against "faithful" IAT under an induced feature gap (removing features exclusively from the VM) on TabbaoAd (Table 1).

Table 1: LoopFM vs. faithful IAT under the same feature gap (TaobaoAd, test AUC, 3 seeds, no KD).

TaobaoAd (VM lacks the 8 user-profile features and 2 behaviour sequences)DeepFMDIENDMRDMIN
Baseline0.5799 ± 0.00010.5846 ± 0.00050.5866 ± 0.00040.5817 ± 0.0003
Faithful IAT0.6214 ± 0.00080.6273 ± 0.00010.6271 ± 0.00040.6256 ± 0.0002
LoopFM0.6266 ± 0.00050.6323 ± 0.00040.6329 ± 0.00020.6305 ± 0.0005
LoopFM − faithful IAT+0.0053 ± 0.0004+0.0049 ± 0.0003+0.0059 ± 0.0006+0.0049 ± 0.0007

Remark Faithful IAT = IAT's supervised 32-d compression bottleneck applied to LoopFM's own FM (same architecture and all FM features), so the two sequences differ only in how the FM's representations are compressed. Mean ± sd over the three seeds (for the difference: sd of the per-seed paired differences). Every difference is positive on all three seeds.

Q1.2 The theory seems to apply to IAT too. Yes. The gain decomposition (Theorem 1) is general, and we see this as a strength: it covers IAT as well as LoopFM, and it explains several of IAT's empirical observations. It splits the gain into (1) temporal information from the history, I_temporal; (2) cross-feature information the VM's own features lack, I_cross; and (3) the loss from compression, I_residual, which arises both in the FM's forward pass and in the auto-encoder.

  • (2) Feature gap. IAT reports cross-domain online gains (its Table 6) but does not analyse or isolate where they come from. In our decomposition they correspond to I_cross > 0. LoopFM targets this term directly, and we isolate it with a controlled feature gap as demonstrated in Appendix L.1 and Table 5 in our response to XQ9B, Q3.
  • (3) Compression loss. IAT observes that saving InsEmb after its SIT module hurts downstream performance (its Sec. 4.4). I_residual explains this: representations closer to the prediction head discard more input information. We study the effect systematically across the embedding, hidden, deep and jointly compressed layers (Sec. 5.3.5, Table 6), and joint multi-layer compression performs best.

Our analysis therefore complements IAT's empirical study. It explains when, why, and how reusing historical representations helps.

AuthorsOpenReview ↗

Response to Q2

Q2.1. Limited improvement over raw sequences.

On TaobaoAd a small margin is expected, as the paper notes, "since the VM can learn equally effective embeddings directly from raw ID sequence." Even so, LoopFM outperforms raw sequences. In practice, it’s impossible to encode all item attributes into long sequences and serve them in VM w/o prohibitive feature storage and model inference cost. In a way, LoopFM provides a powerful framework to compress the vast information, and the quality improves as FM improves.

Table 4 of Sec 5.3.4 is the setting most favorable to raw sequences: all four raw ID sequences at full length. With fewer raw ID sequences, LoopFM's gain grows (Table 2).

Table 2: LoopFM's gain grows as the VM sees fewer raw ID sequences (TaobaoAd). DMIN VM, no KD, mean AUC over 3 seeds. "Raw" is the VM with kk raw ID sequences at full length. "LoopFM vs. Raw" is LoopFM gain over Raw. "(Raw + LoopFM) vs. Raw" is the gain from adding LoopFM on top of the raw sequences.

Raw ID sequences in the VMRawLoopFM v.s. Raw(Raw + LoopFM) v.s. Raw
1 (ad group)0.6232+0.0097+0.0106
2 (+ advertiser)0.6284+0.0044+0.0055
4 (+ campaign, price; = Table 4)0.6307+0.0021+0.0046

Likewise, when the VM's raw history cut to the last 5 steps or the most recent half (LoopFM kept at full length): LoopFM alone leads raw sequence by +0.0266 / +0.0085; on top of raw sequence by +0.0257 / +0.0085 (Table 5 of Sec 5.3.4).

More broadly, the public setup understates LoopFM's advantage over raw sequences. LoopFM's value comes from three properties of industry-scale RecSys that the public benchmarks lack:

  1. A feature gap. On TaobaoAd, the FM and VM see the same features and the FM is small, so there is little information the VM cannot learn from its own raw sequences. In industry-scale RecSys, the FM is far larger (O(10^23) FLOPs) and sees far more features than the VM.
  2. Long FM histories. Latency caps the length of a serving VM's raw sequences while upstream FMs consume much longer histories. LoopFM brings this longer, richer history into the VM by compressing each past interaction, across all of the FM's features, into one compact embedding.
  3. Improving FMs. A raw ID sequence stays the same as the FM improves, but LoopFM's gain grows with it: from 0.13% to 0.57% NE across four FM generations (Fig. 2).

In production, the two are complementary. LoopFM is deployed on top of the VM's existing raw ID sequences with significant gain (Fig. 2). Q2.2. Extend the raw-sequence comparison to KuaiVideo and Amazon. We ran the same design on all benchmarks:

  • The VM's raw history is none / last 5 / most recent half / full / item IDs only, each with and without LoopFM.
  • LoopFM always gets its full sequence. No KD is used.

The result is reported in Table 3 below. On all datasets, LoopFM's gain on top of raw history is all positive (significant in 9 of 12), and it is largest when the VM has no raw sequence. On TaobaoAd, the gain grows as the raw history shrinks, from +0.0046 (full) to +0.0392 (none).

Table 3: LoopFM's gain over raw history at different raw-history level (DMIN, test AUC, no KD, 3 seeds, (n.s.): p ≥ 0.05, paired t-test)

VM raw historyTaobaoAdKuaiVideoAmazon
none+0.0392+0.0139+0.0152
last 5 items+0.0257+0.0049 (n.s.)—
most recent half+0.0085+0.0036 (n.s.)—
full (=paper setting)+0.0046+0.0066+0.0002 (n.s.)
item IDs only+0.0106+0.0096—
mean raw history length (test)49 behaviors, 23 ad IDs97 items7.7 reviews

More results about raw sequence (Appendix L.2)

  • TaobaoAd (Table 5 in Section 5.3.4 and Table 18) LoopFM's lead over raw sequences grows from +0.0021 (full history) to +0.0266 (last 5) as the VM's raw history shrinks. Table 18 repeats the raw sequence study on all six VMs: on DMIN, DIEN and DMR, adding LoopFM on top of raw sequences helps for every fewer- and shorter-raw-sequence setting (all significant).
  • KuaiVideo (Table 19) The LoopFM gain over raw ID sequence holds on DMIN, DIEN, DMR and DeepFM, with 1 / 2 / 4 raw sequences and with the raw history cut to 5 / 10 / 20 steps; every cell is positive; see details in Table 19 of Appendix L.2,
  • Amazon (Table 17) Amazon users have only 7.7 past reviews on average, against 49 on TaobaoAd and 97 on KuaiVideo. Therefore, VM can consume most information directly from raw sequences. In addition, there is no FM-VM feature gap. A summary of the user's own history, whether from LoopFM or IAT, therefore has little to add over the VM's short raw history. LoopFM, however, can group FM embeddings by any key. Keyed by the target item, its sequence holds the FM embeddings of other users' past interactions with that item, information the VM's raw history does not contain. On top of the full raw history, item-keyed LoopFM adds +0.0029 to +0.0065 AUC on all four VMs (5 seeds, every seed positive; see details in Appendix L.3).

Thank you for the detailed response and additional clarifications. I appreciate the authors' efforts in addressing my concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

TL;DR. We thank the reviewers for their constructive feedback. Their questions fall into four categories:

  1. novelty relative to IAT (xgN4, XQ9B);
  2. the gain over raw history sequences, and its consistency beyond TaobaoAd (xgN4, XQ9B);
  3. whether the feature-gap benefit is tested, given the small public FM, and whether FM scale matters (XQ9B); and
  4. the theory and Fig. 2 (Yb6L).

To address them, we ran new controlled experiments on all three public benchmarks (3–5 seeds, paired tests), collected in a new Appendix L, and uploaded a revised PDF.

1. Novelty relative to IAT.

  • Independent work with IAT arXiv preprint: We acknowledge that IAT shares a similar high-level idea of representing each instance as an embedding and grouping them into sequence features. But we are independent works, LoopFM has been deployed in industry-scale recommender systems for two years, through multiple iterations (Sec. 5.4), and delivered +0.5%, +1.03% and +1.22% conversion improvements in three launches.
  • Our contribution compared to IAT paper include (1) theoretical analysis (2) embedding compression using auto-encoder with stop-gradient whereas IAT use bottleneck layers that actually hurts “base model” (3) we provided additional insight and ablation study on layer selection aligning with theoretical analysis. (4) We conducted comprehensive experiments and ablation studies on public datasets, but IAT arXiv paper doesn’t have public data experiments. (5) LoopFM is motivated by industry-scale FM->VM knowledge transfer, especially when there is large capacity and feature gap (e.g., in terms of feature counts or sequence length), and also a general framework that can use any group key (e.g., item ID).
  • Finally, IAT was posted to arXiv in April 2026 and, to the best of our knowledge, has not been peer-reviewed. The ICLR 2027 reviewer guidelines (https://iclr.cc/Conferences/2027/ReviewerGuidelines) state that "authors are not required to compare to papers solely on arXiv," and that "the lack of such comparisons cannot be a basis for rejection." We therefore ask that IAT be treated as concurrent work. We nonetheless cite it and compare against it as above.

2. Gain over raw history sequences, and consistency beyond TaobaoAd. LoopFM improves on raw sequences on all three datasets, and the improvement grows as the VM's raw history becomes shorter or keeps fewer attributes (as reported in Table 5 in Sec 5.3.4 and Appendix L.2), a more realistic industry-scale RecSys setting; Encoding all item attributes into long sequences would incur prohibitive feature storage and VM inference cost in practice. When FM and VM see the same features and the FM is small, a VM can learn much of the same information directly from raw ID sequences hence only a small margin as we saw. Even so, adding LoopFM on top of the full raw history gives a significant gain on all three datasets.

  • TaobaoAd and KuaiVideo: the gain is positive at every raw-history level and rises as the history shrinks, up to +0.0392 and +0.0139 when the VM has no raw history.
  • Amazon: users have only 7.7 past reviews on average, so VM can easily consume raw sequences' information and a summary of the user's own history has little to add. Keying LoopFM by the target item instead (see 1) adds a significant gain on all four VMs.

Our multi-seed reruns of the paper settings further show that the gains are consistent beyond TaobaoAd (see our response to Reviewer XQ9B, Q4). The shorter-history/fewer attributes result is the most relevant to industry-scale systems. There, latency caps the length of the VM's raw sequences while upstream FMs consume much longer histories. This is the setting in which LoopFM is deployed in production, on top of the VM's existing raw ID sequences.

3. Feature gap and FM scale. The feature-gap benefit is tested directly (see Appendix L.1), and we show that LoopFM's gain scales with FM quality. When the VM lacks features the FM sees, LoopFM from a trained FM beats the same sequence from an untrained FM by +0.0057. The gain therefore comes from the FM's knowledge, not merely from having a sequence. FM scale is harder to test on public data: with only four days of training data, larger FMs are worse, not better. Even so, the VM's LoopFM gain rises and falls with FM quality. In production, each FM generation is trained with more compute on large, rich data, and LoopFM's gain grows from 0.13% to 0.57% NE across four generations.

4. Theory and Fig. 2. We agree with all three points and have revised the paper:

  • Derivation. The BCE excess-risk envelope (A1) is now stated as an explicit assumption, not a derived result, and Appendix F.7 is reworded to match.
  • Eq. (8). Theorem 2(2), Corollary 4 and Appendix F.8 now include the nonnegativity condition.
  • Fig. 2. A power law fits the four points well (R² = 0.992). We redraw the figure with this fit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.