- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
LoopFM transfers intermediate information from a recommendation FM to compact serving models alongside scalar KD. It compresses offline FM activations with an autoencoder and organizes past-interaction embeddings into temporally ordered user-history features, avoiding FM inference during serving. The population analysis separates temporal-history information, additional FM-derived information and a pipeline residual, alongside a conditional transfer-ratio analysis. Public experiments show AUC gains over KD on TaobaoAd, with more mixed results elsewhere, and industrial deployments report conversion gains. Comparisons with simpler history and embedding-transfer alternatives assess the pipeline's incremental value, while representation, compression and encoder studies examine practical design choices.
Critical strengths
-
The staged pipeline description makes the offline extraction, compression and serving interface easy to follow. The experimental questions connect the design choices to specific comparisons.
-
Offline storage of past-interaction embeddings provides a practical interface for compact serving models. The compression and encoder studies address the storage and capacity constraints of that design.
-
The population information decomposition usefully distinguishes raw historical signal from additional FM-derived information and residual history information. This gives the history-based comparisons a clear analytical context.
-
TaobaoAd AUC gains across student architectures and a repeated-seed check for DeepFM support the method's empirical value. Reported production conversion gains add evidence of deployment utility.
-
Strong pooling, raw-history and item-only baselines help assess the incremental value of the full pipeline. Layer and checkpoint comparisons also reveal practical tradeoffs in choosing representations.
Critical weaknesses
-
A1 in Sec. 4.2 gives Eq. (6) that presented as a consequence of NTK and benign overfitting, but Appendix F.7 does not establish the required BCE excess-risk envelope: classification-decision equivalence does not control BCE risk, and the stated benign limits do not establish the relative rate needed to absorb the bias term. Treating the BCE envelope as an independent assumption would preserve the conditional analysis; the claimed derivation needs a loss-specific proof or narrower wording.
-
Eq. (8) in Theorem 2(2) omits the nonnegativity condition on the numerator's lower bound stated in Appendix F.4. Assumption A3 does not imply this condition, which is needed for the proof's denominator substitution. Please state it in the main theorem.
Questions for authors
- What fitted relation and predictive check support the power-law claim in Fig. 2, and which NE reference and fixed teacher, data, compression and student settings define the comparison? Otherwise, please describe the evidence as an observed trend.
2 comments
Thank you for the clarifications and revisions, which address my main concerns. I will therefore raise my Soundness score from 2 to 3 while retaining my weak acceptance. I have no further questions.
We thank the reviewer for the careful reading of the theory.
Q1. A1 / Eq. (6) is presented as derived, but Appendix F.7 does not establish the BCE excess-risk envelope.
We agree. The NTK and benign-overfitting results we cite give classification-decision equivalence and benign limits for the linearized model. They do not give a two-sided BCE excess-risk envelope at the relative rate needed to absorb the bias term.
The revision states the two-sided envelope in Eq. (6) as an explicit assumption (A1), in Sec. 4.2 and Appendix E. Appendix F.7 is retitled as motivation under the stated NTK and benign-overfitting conditions, rather than a derivation, and its cross-entropy paragraph no longer claims that Bartlett's bounds transfer to the BCE excess risk. The corollary in F.7 that recovers the envelope is now stated for the linearized squared-loss surrogate only, and the general-covariance bound (Appendix F.8) is stated with the assumed envelope. Appendix F.7's corollary now states that the linearized (NTK / benign-overfitting) result gives the two-sided envelope only for the linearized squared-loss surrogate; for the BCE loss the envelope is an assumption (A1), not a derived result. As the reviewer notes, all downstream results (Theorem 2, Corollary 4) are conditional on this envelope and remain valid as conditional statements; the revision says so in Sec. 4.2 and in the Limitations.
W2. Eq. (8) in Theorem 2(2) omits the nonnegativity condition from Appendix F.4.
The proof substitutes the denominator bound, which requires the numerator's lower bound to be nonnegative. A3 does not imply this, and it is necessary: if the numerator's lower bound is negative, dividing it by an upper bound on the denominator no longer gives a valid lower bound. The revision adds it to the statement of Theorem 2(2) (as condition (⋆), so the existing equation numbers are unchanged), to Corollary 4 and to the general-covariance version of the bound (Appendix F.8), and notes that it holds whenever the feature-gap term dominates the temporal pipeline loss, i.e. δ ≥ δ_min = τ₂ I_temporal / ((1 − η₁) κ^hist_gap).
Q. Fig. 2: what fitted relation and predictive check support the power-law claim? Which NE reference and fixed settings define the comparison?
Fit on Fig. 2's four plotted points (compute 0.84 / 2.00 / 2.90 / 5.00 × 10²³ FLOPs → NE gain 0.13 / 0.24 / 0.37 / 0.57%):
The power law fits better and predicts the held-out point within 0.02 percentage points. In the revision we:
Settings. Each point in Fig. 2 is taken from one LoopFM launch on the same production VM model, distilled from a different version of the FM; we report each FM version's training compute (FLOPs) and LoopFM's relative NE gain. Unfortunately, we do not have enough time within the rebuttal period to obtain legal approval to release the detailed setup of these internal results. We hope the fitted relation and the held-out check address the core of the question, and the revision softens the wording as described.