- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The submission investigates the selection among multiple candidate adapters to be plugged into a deployed multimodal recommender before fully adapting it. All models (image, text, fusion, joint residual branches) initially produce the same predictions. The authors propose a new "correction score" to predict the compatibility of an updater with the reference residual using ridge regression of the residual on a 64-dimensional sketch of an updater Jacobian initialized on one user group. They emphasize the equivalence of their proposal to an offset residual linear probe. The novelty lies in applying it as a selection criterion. In the experimental evaluation on Amazon Baby, MicroLens-1M and Pixel200K, the method was compared against development block priors, transferability scores, and short Direct-k trials. The authors state superior updater ranking compared to Direct-10, no difference from Direct-60, a two-stage screen-then-trial search that preserves 96% of the oracle gains at 40% of the cost, prediction of Brier/NDCG gains, and a study of preservation under replay vs. task loss constraint conditions.
Critical strengths
The core intuition is sound: the same reference error is not equally fixable by every adapter. The authors are honest that the score is a known linear probe, and they compare it against the right baselines, a fixed development-block ranking and short training runs with the same budget.
Critical weaknesses
i. The results cannot be verified. The submission states that the supplement lets you reproduce the analyses, but it does not. The run scripts (run_primary.py, run_screening.py, run_preservation.py) and the preprocessing scripts only print a message. Most of the pipeline is missing: training the reference, extracting features, sampling comparison items, NDCG evaluation, and the FREEDOM/BM3 experiments. The data folder holds copies of the paper’s tables and no split files. The code (that does exist) does not always match the paper. For example, roles are assigned by hash rather than by the early/late split described in Appendix B.2, and the Jacobian is computed on a single item rather than on the pair probability (C.2). Since the whole protocol is custom, there is no way to check any number in the paper.
ii. It is unclear whether the problem matters in practice. The reference appears to be trained once on the first 40% of the data and never updated, which probably makes it look more correctable than a real production model would be. There is also no comparison with simply fine-tuning or retraining it. The evidence rests on small cohorts (48 users to fit, 192 to evaluate) and a single random user split. Several key gains have intervals that barely exclude zero, such as +2.9 over Direct-10. NDCG differences between adapters are tiny, and training all 12 candidates fully takes about 27 seconds, so the efficiency argument is hard to buy at this scale.
Questions for authors
i. Is the reference ever retrained or refreshed across the 30 blocks? How does the score’s advantage vary with block index? ii. Please, release the code that produced the tables, plus condition-level outputs (per block × group × updater) and the exact split artefacts. Which implementation of the role allocation and Jacobian produced the reported results? iii. How do results vary across several random user-to-role allocations? iv. How does adapter selection compare with simply fine-tuning or retraining the reference on recent data? v. Why does the supplement’s Table 9 (78.4/76.9/78.1) differ from the paper’s one (80.4/79.1/80.0)?