- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Authors introduce ISCon, a framework for testing whether personalization models use user histories according to their stated design goals, such as prioritizing recent interests or preserving long-term preferences. Its testing procedure, Prefturb, isolates, moves, removes, or repeats preference (i.e., perturbation) evidence and measures how outputs change.
Critical strengths
None
Critical weaknesses
The paper is NOT readable. Some examples:
(Introduction) Metamorphic and behavioral testing compare responses across controlled inputs against specified expectations. History shuffling has also been used to examine whether LLM recommenders use interaction order. Order sensitivity alone cannot establish adherence to a particular personalization rationale.
(Table 4 caption) RQ3 conflict controls. Mamba4Rec, MovieLens. MRR \times 100. Descriptive Endpoints.
- what is ‘conflict controls’? What are ‘descriptive endpoints’?
(Section 5, Experimental setup) Reanalysis uses saved outputs, with no training or new model inference.
- Saved where? This is not a language to use in the paper, this is a language when coding agents generate reports to the users.
(Section 7, Limitations) Output comparisons also do not identify internal mechanisms. Construction may fail to produce a valid test for some histories. Empirical intervals are exploratory and conditional on saved configurations. The formal error guarantee requires its stated sampling and coverage assumptions.
- what does it mean that output comparisons do not identify internal mechanisms? What is the internal mechanism referring to? What construction is failing, what are the valid and invalid tests? What is the definition of exploratory intervals and what does it mean ‘saved configurations’, where is it saved, is it saved on some storage?
These are just some examples. The entire paper is impossible to follow and written in a very non-human style. Constructive review is possible only if the authors provide a sufficiently readable manuscript.
Questions for authors
Please refer to Weaknesses