- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates the interaction between user context and test-time compute in personalization. Through experiments on LaMP, this paper demonstrates that user context generally provides more substantial and reliable performance gains and further shows that the interaction between user context and model scale varies depending on the task.
Critical strengths
None
Critical weaknesses
-
The experimental design needs to be revised:
- First, regarding the experiments that increase the amount of user context, rather than simply increasing the value of k in BM25, the authors should first identify the oracle user contexts for each task and then gradually increase the number of oracle user contexts provided. Since the experiments were not conducted under such an oracle setting, and potentially irrelevant user contexts could be included, there is a fundamental flaw in the paper's assumption that "providing more user context" necessarily means providing more relevant information.
- Furthermore, although the paper claims to investigate the effects of test-time compute scaling, the experiments simply compare different model tiers within the same model family. Changing the model itself does not constitute test-time compute scaling. To properly investigate test-time compute scaling, the authors should have varied parameters such as reasoning effort within the same model.
-
Use of a limited and outdated benchmark:
- Although many recent personalization benchmarks have been introduced across diverse tasks with more realistic settings, this paper evaluates its approach only on LaMP, which has outdated and limited characteristics for evaluating the personalization capabilities of modern advanced systems (e.g., web agents, search agents, etc.). Of course, given the nature of this work, it may not be necessary to conduct experiments across a wide range of benchmarks. However, I believe that including experiments on at least some recent personalization benchmarks would further strengthen the value of this work.
Questions for authors
Refer to weakness