- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Since most performance metrics for recommendation focus only on user interaction, the authors propose in this paper a new performance metric, the content depth score (CDS), which combines three cognitive theoretical anchors: Dual Process Theory (Wason & Evans, 1974), the Bloom’s Taxonomy (Anderson & Krathwohl, 2001), and the SOLO Taxonomy (Biggs & Collis, 2014). The authors then present SCOPE-Bench, the first benchmark for CDS. To efficiently produce CDS benchmark, the authors first construct a gold-standard subset through human annotation, and then conduct a series of analyses to show that employing LLMs is able to obtain annotations of quality close to human annotations, thereby facilitate automatic annotation. In addition, the authors propose two CDS-based performance metrics for recommendation.
Critical strengths
This paper presents the first benchmark for CDS, and shows the possibility to employ LLMs to generate CDS annotations of quality close to human annotations.
Critical weaknesses
None
Questions for authors
It is possible to simply revise traditional utility-based performance metrics, such as NDCG, to support CDS. For example, we can obtain a CDS version of NDCG by substituting relevance scores in NDCG with CDS scores. Please clarify the strengths of A-LCDS and E-LCDS.
3 comments
Follow-up: summary of our responses
Thank you again for your positive assessment and your constructive question. With the discussion period nearing its end, we briefly restate our answer.
Compared with a CDS-based NDCG, A-LCDS and E-LCDS use the highest CDS level as a fixed reference instead of the ideal ranking within each candidate set. This design has two advantages.
- Interpretability: The scores map directly onto the CDS scale. With the default , 6 × A-LCDS@K equals the mean CDS of the list, and 6 × E-LCDS@K equals its position-weighted mean CDS.
- Absolute depth: The scores preserve differences in depth across candidate sets. If every candidate has a CDS of 1, an ideal ranking still achieves a CDS-NDCG of 1, whereas both LCDS metrics equal 1/6 and correctly reflect the low depth of the list.
We will add this discussion to the final version. We would appreciate your taking a look at our rebuttal. If it resolves your concerns, please consider revising your score upward. We would be happy to clarify any remaining questions.
Gentle Reminder: Discussion Period Ends in 12 Hours
Dear Reviewer zpFU,
Thank you again for your positive assessment, and we apologize for following up again. With less than 12 hours left in the discussion period, we would like to check whether our answer has addressed your question.
In brief, A-LCDS and E-LCDS use the highest CDS level as a fixed reference instead of the ideal ranking within each candidate set. As a result, (1) the scores map directly onto the CDS scale, and (2) they preserve absolute differences in depth across candidate sets, which a CDS-based NDCG would normalize away. We will add this discussion to the final version.
If you have any further questions, please let us know and we will respond promptly. If our answer has resolved your question, we would sincerely appreciate it if you could consider raising your score.
Best regards,
The Authors
Response to Q: Advantages of A-LCDS and E-LCDS over CDS-NDCG
Thank you for your constructive comments and for your recognition of our paper.
The main differences between CDS-NDCG and our metrics lie in the interpretation of the resulting scores. Specifically, CDS-NDCG evaluates a recommendation list against the ideal depth-based ranking within the candidate set, whereas A-LCDS and E-LCDS adopt the highest CDS level as a fixed reference. This design maintains a direct correspondence between list-level scores and the original CDS scale. Under the default setting, g(s)=s/6, and thus A-LCDS@K multiplied by 6 equals exactly the average CDS of the recommendation list. For example, an A-LCDS of 0.5 indicates that the average CDS of the list is 3. E-LCDS further accounts for ranking positions by applying a position discount that assigns higher weights to top-ranked videos. Accordingly, E-LCDS multiplied by 6 equals the position-weighted average CDS. Therefore, the two metrics provide intuitive interpretations of the average content depth and the position-weighted content depth, respectively.
In addition, the fixed reference preserves differences in content depth across candidate sets. For instance, when all candidate videos have a CDS of 1, the ideal ranking still achieves a CDS-NDCG of 1, whereas both LCDS metrics equal 1/6, which correctly reflects the low depth of the recommended content.
In summary, the main strengths of A-LCDS and E-LCDS lie in their interpretability, which stems from their direct correspondence with the original CDS scale, and in their complementary characterization of the average content depth and the effect of ranking position.