- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how text-profile recommendation systems should respond to natural-language user requests, focusing on both the direction and magnitude of preference changes. The authors propose Calibrated Edit Distillation (CED), which uses population-level preference odds ratios to construct calibrated score adjustments and distills these adjustments into a recommendation model without requiring a separate parser. Experiments on MovieLens-1M and MovieLens-20M compare CED against standard ranking models, direction-only training, and other request-following approaches. The results demonstrate improvements in handling preference insertion, withdrawal, and negation, including generalization to unseen attributes and request phrasings, with limited observed degradation in recommendation accuracy.
Critical strengths
The main strength of this paper is its distinction between following the direction of a user request and responding by an appropriate amount. CED provides a principled calibration target based on population preference statistics and integrates it into the recommendation model through distillation. The experiments include multiple baselines, unseen attributes, and unseen request phrasings, providing useful evidence of the method's effectiveness and generalization ability.
Critical weaknesses
The main limitation is that the proposed calibration target relies on population-level preference statistics, which may not fully represent the appropriate response magnitude for individual users. Although the paper includes an evaluation of users with changing preferences, the general validity of this calibration assumption remains insufficiently established. In addition, the performance under unseen request phrasings is less consistent than the main results suggest. For example, in Table 11, CED produces a recommendation share of 0.15 for withdrawn preferences compared with the reference value of 0.22, indicating that the model may still overreact to certain requests. Finally, the claim of no measurable accuracy degradation is based on only three training seeds, with relatively large variation in NDCG@10 changes, making the stability of this conclusion less certain.
Questions for authors
- The calibration target is derived from population-level preference statistics. Could the authors further clarify how this target reflects the appropriate response magnitude for individual users? Could the existing evaluation results be used to examine variation in calibration errors across users or attributes?
- In Table 11, the response to unseen withdrawal phrasings remains below the reference level (0.15 versus 0.22). Could the authors clarify why calibration becomes less accurate for these requests and whether this behavior is consistent across different attributes?
- The reported NDCG@10 changes show relatively large variation across three training seeds. Could the authors further clarify the stability of recommendation accuracy using the existing per-seed results and discuss whether the current evidence is sufficient to support the claim of no measurable accuracy degradation?