- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The author proposed a system targetting individualize recommendation to specific researchers with limited training data. The topic is novel and interesting as existing work mostly focuses on general comparisons between ideas. The author conducted experiments comparing their system with several baselines and also decomposed each component of the system to give the reader a better understanding of the paper. Through experiment, the author found that there can be large divergence between personal taste and general taste. The author also deployed this system with 20 real researchers from multiple domains to show the effectiveness of the system.
Critical strengths
Firstly, the topic of individualize recommendation is novel and impactful. Existing work mostly focuses on capturing the general taste or trend but overlook the fact that different researcher will have significantly different perspetives.
Second, the proposed pipeline is easy to deploy and adapt, as it requires no parameter training.
Critical weaknesses
Firstly, the claim of "individualize" proves having limited benefit on the author proposed assessment. On table 6, adding the individualize loop only gives a limited 0.013 gain, which can be attributed to noise. The major claim of the author is thus not convincing.
Secondly, the advantage of the proposed pipeline can also be attributed to the strength of the backbone model instead of the success of the pipeline. From Table 1 and Table 2, it seems the proposed pipeline relies on GPT-5.5 or other proprietary models, but competitive baselines such as VPL or FSPO rely on much smaller and older models such as Qwen2.5-1.5B-Instruct or GPT-2. And from Table 2, after swithing the backbone model to GPT-5.4-mini the performance is already worse than FSPO based on Qwen2.5-1.5B-Instruct (the result of FSPO is in Table 1). And from my perspective, GPT-5.4-mini is stronger than Qwen2.5-1.5B-Instruct.
Thirdly, the sample number is too small with only about 70 real user data points. The reported numbers don't open up large gaps, so many advantage can be attributed to noise instead of system level success.
Questions for authors
-
About the design choices, the author chose six criteria such as generality, novelty, importance, parsimony, depth, and algorithmic complexity. How did you make this choice? Is that based on some existing literature or some experiments? When later switching to domain such as computer science, statistics, and medicine, do these criteria change tailored to the specific domain requirements?
-
How do you control training data contamination in the model? The pipeline relies on published papers so the backbone LLM has very likely seen the papers during training. Are those papers published after the training cutoff of the evaluated models?
-
Further ablation is needed to make the pipeline to be more convincingly useful. For example, could the authors provide a per-researcher logistic regression on the same six criterion scores without any population prior or any rules? Also, for FSPO, the original paper uses a stronger backbone Llama-3.2-3B, could the authors provide a result based on the original implementation of FSPO?