acceptodds
Withdrawn from ICLR 2027

Screening Updaters for Multimodal Recommender Adaptation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A recommender often starts from one deployed reference model but still has several ways to update it. Those candidate updaters share the same reference predictions, yet they may not be able to correct the same part of the reference error. We study which updater is worth trying when several updater mechanisms share one reference. A held-out local correction score fits the reference residual in a finite tangent sketch on one earlier cohort and evaluates that learned direction on disjoint users before full adaptation. The score is not a new linearization estimator; it is a screening signal for same-reference updater selection. Across three multimodal recommendation sources, it improves on fixed development rankings, adapter-aware transfer baselines, and short nonlinear trials at the primary scale. Supporting 24- and 48-candidate searches show where screening becomes useful as candidate count or updater scale increases: the score narrows the search before a shortlist receives explicit nonlinear trials. The same measurement also predicts later Brier and ranking gains across time blocks. Preservation is a downstream decision after updater selection: replay keeps more average improvement, while an explicit task-loss requirement reduces positive group degradation when measured correction is large. Capacity, locality, and timing checks define the setting in which this staged updater-search procedure is most useful.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Recommendation, Recommender Adaptation, Updater Selection, Parameter-Efficient

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YEUxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The submission investigates the selection among multiple candidate adapters to be plugged into a deployed multimodal recommender before fully adapting it. All models (image, text, fusion, joint residual branches) initially produce the same predictions. The authors propose a new "correction score" to predict the compatibility of an updater with the reference residual using ridge regression of the residual on a 64-dimensional sketch of an updater Jacobian initialized on one user group. They emphasize the equivalence of their proposal to an offset residual linear probe. The novelty lies in applying it as a selection criterion. In the experimental evaluation on Amazon Baby, MicroLens-1M and Pixel200K, the method was compared against development block priors, transferability scores, and short Direct-k trials. The authors state superior updater ranking compared to Direct-10, no difference from Direct-60, a two-stage screen-then-trial search that preserves 96% of the oracle gains at 40% of the cost, prediction of Brier/NDCG gains, and a study of preservation under replay vs. task loss constraint conditions.

Critical strengths

The core intuition is sound: the same reference error is not equally fixable by every adapter. The authors are honest that the score is a known linear probe, and they compare it against the right baselines, a fixed development-block ranking and short training runs with the same budget.

Critical weaknesses

i. The results cannot be verified. The submission states that the supplement lets you reproduce the analyses, but it does not. The run scripts (run_primary.py, run_screening.py, run_preservation.py) and the preprocessing scripts only print a message. Most of the pipeline is missing: training the reference, extracting features, sampling comparison items, NDCG evaluation, and the FREEDOM/BM3 experiments. The data folder holds copies of the paper’s tables and no split files. The code (that does exist) does not always match the paper. For example, roles are assigned by hash rather than by the early/late split described in Appendix B.2, and the Jacobian is computed on a single item rather than on the pair probability (C.2). Since the whole protocol is custom, there is no way to check any number in the paper.

ii. It is unclear whether the problem matters in practice. The reference appears to be trained once on the first 40% of the data and never updated, which probably makes it look more correctable than a real production model would be. There is also no comparison with simply fine-tuning or retraining it. The evidence rests on small cohorts (48 users to fit, 192 to evaluate) and a single random user split. Several key gains have intervals that barely exclude zero, such as +2.9 over Direct-10. NDCG differences between adapters are tiny, and training all 12 candidates fully takes about 27 seconds, so the efficiency argument is hard to buy at this scale.

Questions for authors

i. Is the reference ever retrained or refreshed across the 30 blocks? How does the score’s advantage vary with block index? ii. Please, release the code that produced the tables, plus condition-level outputs (per block × group × updater) and the exact split artefacts. Which implementation of the role allocation and Jacobian produced the reported results? iii. How do results vary across several random user-to-role allocations? iv. How does adapter selection compare with simply fine-tuning or retraining the reference on recent data? v. Why does the supplement’s Table 9 (78.4/76.9/78.1) differ from the paper’s one (80.4/79.1/80.0)?

Reviewer_uSYVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to select update modules for a fixed multimodal recommender. It uses a held-out correction score based on linearized updater features to shortlist candidates for short training trials. Experiments on three recommendation datasets evaluate updater selection, prediction of future gains, and search costs. The reported results suggest that screening can reduce search costs while retaining most of the gains available from the candidate set.

Critical strengths
  1. The paper addresses a practical problem: selecting an update module without fully training every candidate. The reported results suggest that screening followed by short training trials can retain most of the best candidate’s gains at lower search cost.

  2. The main experiments compare modules with identical initial predictions and parameter counts across three datasets. Separate fitting and evaluation users, short-training baselines, and paired block confidence intervals make the evidence for improved updater selection more convincing.

Critical weaknesses
  1. The methodological novelty is limited. As acknowledged in Section 2 and Appendix A.2, the correction score is equivalent to an offset residual ridge probe in a sketched tangent space. The contribution therefore rests mainly on its empirical value for updater selection. However, Appendix C.3 does not clearly explain how the empirical NTK baseline differs in its regression target, reference offset, regularization, or scoring procedure. This makes the reported advantage over a closely related method difficult to assess.

  2. The efficiency comparison with successive halving is not fully specified. The reported costs are derived from standalone trial timings (Tables 21 and 34), but it is unclear whether the step budgets are cumulative and whether surviving candidates reuse their model and optimizer states. If training resumes under cumulative budgets, adding full trial times could overcount computation. This leaves uncertainty about the efficiency advantage over successive halving, an important comparison given the paper’s focus on reducing search cost.

Questions for authors
  1. Could you clarify how the empirical NTK regression baseline differs from the correction score in its kernel or sketch, regression target, reference offset, regularization, and held-out scoring? Which differences account for the performance gap reported in Table 1?

  2. For successive halving, do the 1, 2, 5, and 10 steps represent cumulative budgets or additional updates at each stage? Are model and optimizer states reused between stages? Please explain how the reported search costs account for this. If candidates are restarted, could you provide a timing comparison that resumes training from previous checkpoints?

Reviewer_XSe3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to choose among update mechanisms attached to the same multimodal recommender before full adaptation. It fits reference residuals in a low-dimensional tangent space and evaluates the learned correction on disjoint users to rank candidate updaters. Experiments examine updater selection, search efficiency, and the subsequent trade-off between adaptation gains and preservation.

Critical strengths
  1. Choosing which branch to update is a relevant maintenance problem. Holding the reference predictions and parameter budget fixed makes the comparison easy to interpret. This design isolates updater choice from differences in initial model quality.

  2. The experiments include meaningful controls, such as development priors, Jacobian-based probes, NTK regression, and short nonlinear trials. The larger candidate searches also compare against successive halving. These comparisons provide evidence that the score can be useful for narrowing a candidate set.

  3. The evaluation protocol carefully separates users across fitting, scoring, and future evaluation. The locality and temporal analyses help clarify when the score becomes less reliable. The manuscript also documents implementation details and reports anonymized supplementary code and configurations.

Critical weaknesses
  1. The methodological novelty is limited. The correction score is a held-out residual ridge probe on tangent features, and its interpretation follows from a standard projection identity. Applying this estimator to updater selection appears incremental, while the screening-then-trial workflow follows established low-cost model selection practice.

  2. Several closely related PEFT studies are missing. Xu et al. (2026) already use projected residuals to guide adapter placement, Afzal et al. (NeurIPS 2025) study NTK-based prediction and selection of adaptation configurations, and BIPEFT (Findings of EMNLP 2024) addresses efficient budget-guided module search. These connections substantially narrow the claimed distinction of same-reference updater selection and require a clearer discussion of what remains new.

  3. The gains over short nonlinear trials are relatively modest: 2.9 percentage points in pair ordering and 1.9 points when ordering by future NDCG gain. These are selection metrics, and the paper does not directly establish the final recommendation improvement over competing search strategies under the same total compute budget. The catalogue results compare preservation objectives rather than the proposed selector.

  4. The practical significance of the efficiency gains is not fully established. In the primary twelve-candidate experiment, search time falls from approximately 0.94 to 0.38 seconds with content features already cached. The reported search savings exclude the selected updater’s full training, so a complete cost–performance comparison is needed to show how much the workflow improves an actual update cycle.

  5. The evidence remains concentrated on frozen content encoders and centered residual updaters, including the larger-capacity experiments. It is unclear whether the selection advantage persists with other common update parameterizations or more substantial distribution changes. This restricted setting also limits the broader insight beyond the expected observation that local linearization becomes less reliable for larger updates.

Questions for authors
  1. What does the proposed score contribute beyond an existing held-out residual probe, and what technical difficulty is introduced specifically by same-reference updater selection?

  2. How does the method differ from projected-residual layer selection, NTK-based PEFT evaluation, and budget-guided PEFT search? Can the closest approaches be compared within the same candidate space?

  3. Under a matched total compute budget, how much does the selected model improve final NDCG and Recall over short trials and successive halving?

  4. What are the total update-cycle costs, including screening, shortlist trials, and the selected updater’s full training?

  5. Does the advantage persist with conventional LoRA parameterizations and stronger temporal shifts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.