acceptodds
Under review as a conference paper at ICLR 2027

Content Depth Matters in Short-Video Recommendation: Rethinking the Attention Economy

Abstract

Driven by the attention economy, short-video Recommender Systems (RSs) are primarily optimized to maximize user engagement by promoting videos that capture attention within seconds. These systems inherently favor shallow-content videos that are effective at attracting immediate attention. However, growing evidence suggests that prolonged exposure to such content may negatively affect users' cognitive engagement and mental well-being, raising concerns about the long-term societal impact of the short-video platform. A natural extension of the current attention-centric recommendation system is to incorporate additional metrics that promote user well-being. Toward this goal, this paper introduces a new metric, the Content Depth Score (CDS), to quantify the content depth of short videos. It measures the extent to which a video is expected to stimulate higher-order cognitive processes, using a seven-level scale grounded in established theories of cognitive psychology and learning. As an initial step toward this vision, we present SCOPE-Bench, the first benchmark for content-depth evaluation in short-video recommendation. Built upon a large-scale open-source short-video dataset, SCOPE-Bench provides CDS annotations for 150K videos, enabling systematic evaluation of RSs from a cognitive-content perspective. Leveraging SCOPE-Bench, we evaluate 13 representative RSs and reveal a consistent preference for shallow-content videos. Moreover, we find that these algorithms recommending cognitively deep content are only marginally better than random selection, highlighting a previously overlooked limitation of existing recommendation objectives. The code and datasets are available at the link below.

Primary area:
datasets and benchmarks
Keywords:
Benchmark and Datasets, Recommender Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zpFUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Since most performance metrics for recommendation focus only on user interaction, the authors propose in this paper a new performance metric, the content depth score (CDS), which combines three cognitive theoretical anchors: Dual Process Theory (Wason & Evans, 1974), the Bloom’s Taxonomy (Anderson & Krathwohl, 2001), and the SOLO Taxonomy (Biggs & Collis, 2014). The authors then present SCOPE-Bench, the first benchmark for CDS. To efficiently produce CDS benchmark, the authors first construct a gold-standard subset through human annotation, and then conduct a series of analyses to show that employing LLMs is able to obtain annotations of quality close to human annotations, thereby facilitate automatic annotation. In addition, the authors propose two CDS-based performance metrics for recommendation.

Critical strengths

This paper presents the first benchmark for CDS, and shows the possibility to employ LLMs to generate CDS annotations of quality close to human annotations.

Critical weaknesses

None

Questions for authors

It is possible to simply revise traditional utility-based performance metrics, such as NDCG, to support CDS. For example, we can obtain a CDS version of NDCG by substituting relevance scores in NDCG with CDS scores. Please clarify the strengths of A-LCDS and E-LCDS.

3 comments
AuthorsOpenReview ↗

Response to Q: Advantages of A-LCDS and E-LCDS over CDS-NDCG

Thank you for your constructive comments and for your recognition of our paper.

Q: Advantages of A-LCDS and E-LCDS over a CDS-based variant of NDCG

The main differences between CDS-NDCG and our metrics lie in the interpretation of the resulting scores. Specifically, CDS-NDCG evaluates a recommendation list against the ideal depth-based ranking within the candidate set, whereas A-LCDS and E-LCDS adopt the highest CDS level as a fixed reference. This design maintains a direct correspondence between list-level scores and the original CDS scale. Under the default setting, g(s)=s/6g(s) = s/6, and thus A-LCDS@K multiplied by 6 equals exactly the average CDS of the recommendation list. For example, an A-LCDS of 0.5 indicates that the average CDS of the list is 3. E-LCDS further accounts for ranking positions by applying a position discount that assigns higher weights to top-ranked videos. Accordingly, E-LCDS multiplied by 6 equals the position-weighted average CDS. Therefore, the two metrics provide intuitive interpretations of the average content depth and the position-weighted content depth, respectively.

In addition, the fixed reference preserves differences in content depth across candidate sets. For instance, when all candidate videos have a CDS of 1, the ideal ranking still achieves a CDS-NDCG of 1, whereas both LCDS metrics equal 1/6, which correctly reflects the low depth of the recommended content.

In summary, the main strengths of A-LCDS and E-LCDS lie in their interpretability, which stems from their direct correspondence with the original CDS scale, and in their complementary characterization of the average content depth and the effect of ranking position.

AuthorsOpenReview ↗

Follow-up: summary of our responses

Thank you again for your positive assessment and your constructive question. With the discussion period nearing its end, we briefly restate our answer.

Compared with a CDS-based NDCG, A-LCDS and E-LCDS use the highest CDS level as a fixed reference instead of the ideal ranking within each candidate set. This design has two advantages.

  • Interpretability: The scores map directly onto the CDS scale. With the default g(s)=s/6g(s) = s/6, 6 × A-LCDS@K equals the mean CDS of the list, and 6 × E-LCDS@K equals its position-weighted mean CDS.
  • Absolute depth: The scores preserve differences in depth across candidate sets. If every candidate has a CDS of 1, an ideal ranking still achieves a CDS-NDCG of 1, whereas both LCDS metrics equal 1/6 and correctly reflect the low depth of the list.

We will add this discussion to the final version. We would appreciate your taking a look at our rebuttal. If it resolves your concerns, please consider revising your score upward. We would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ends in 12 Hours

Dear Reviewer zpFU,

Thank you again for your positive assessment, and we apologize for following up again. With less than 12 hours left in the discussion period, we would like to check whether our answer has addressed your question.

In brief, A-LCDS and E-LCDS use the highest CDS level as a fixed reference instead of the ideal ranking within each candidate set. As a result, (1) the scores map directly onto the CDS scale, and (2) they preserve absolute differences in depth across candidate sets, which a CDS-based NDCG would normalize away. We will add this discussion to the final version.

If you have any further questions, please let us know and we will respond promptly. If our answer has resolved your question, we would sincerely appreciate it if you could consider raising your score.

Best regards,
The Authors

Reviewer_ygpZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Content Depth Score (CDS), a seven-level rubric for assessing how deeply a short video develops information, from primarily affective content to transferable models and frameworks. It further proposes two list-level metrics, A-LCDS and E-LCDS, for measuring the average and rank-weighted content depth of recommendation list results.

Using CDS, the authors construct SCOPE-Bench by annotating approximately 154,000 videos from the ShortVideo dataset. The automated evaluator is selected by comparing six LLMs against human annotations on an 833-video gold set, with additional evaluation on a 50-video challenging set. Experiments with 13 recommender systems show that, although these methods outperform random recommendation on engagement-oriented metrics, their recommended content remains predominantly low-depth and close to the random baseline in LCDS. CDS-aware training experiments further demonstrate a trade-off between recommendation relevance and content depth.

Critical strengths

1, A useful new evaluation axis with a clear and concrete scoring framework. The paper turns a socially important but vague concern into a documented seven-level rubric, a list metric, and a benchmark. The prompt, boundary rules, examples, null-label treatment, random-baseline derivation, and implementation details are substantially disclosed (Secs. 3–4; Apps. A–D and K). This makes the proposal inspectable and potentially useful even if the exact rubric is revised.

2, Broad and thoughtful empirical validation. The study combines independent human ratings, chance-corrected ordinal agreement with bootstrap intervals, six candidate LLM judges, a deliberately difficult high-depth set, temperature sensitivity, category and verbosity counterfactuals, 13 recommender baselines on two dataset variants, training trajectories, and a CDS-aware optimization demonstration (Secs. 5–6; Apps. C, E–J). The authors also avoid claiming that CDS measures factuality or actual cognitive outcomes (Ethics Statement and App. L, pp. 10 and 43).

Critical weaknesses

1, Construct validity remains incomplete. The results show that humans and the selected LLM can apply the CDS rubric consistently, but they provide less evidence that CDS measures content depth itself rather than related features such as topic, transcript length, writing style, or cues favored by the evaluator. The randomly sampled gold set contains almost no examples at levels 4–6, so the reported aggregate agreement provides limited evidence for the full seven-level scale. The 50-item hard set improves coverage but is small and selectively constructed. Additional validation using independent expert judgments, matched examples that control for topic and transcript length, and sufficient per-level results would strengthen the claim that CDS measures informational structure rather than surface correlates.

2, The recommender conclusions are descriptive but not yet statistically or causally established. Roughly 79.4% of items are either CDS 0 or null (Fig. 5, p. 8), and the main lower-bound metric maps null to zero (footnote 3, p. 6). Thus low LCDS and proximity to a random catalog baseline can largely reflect catalog prevalence and missing ASR; treating null as zero also conflates unobservable depth with shallow content. Valid-only results help, but no confidence intervals or paired significance/effect-size tests are reported for model-vs-random or model-vs-model LCDS differences, despite five seeds. The 44-user case study does not establish that model architecture “causes” low-depth recommendations: it lacks a matched candidate-distribution baseline and does not control popularity, exposure, category, or the feedback-generation process. Recall/NDCG on logged implicit interactions measure offline relevance, not “attention” directly, so claims that engagement optimization drives shallow content should be narrowed unless supported by causal or online evidence. These issues affect the central conclusion rather than only presentation.

Questions for authors

1, Construct validation: Can the authors provide additional validation showing that CDS measures content depth rather than related features such as topic, transcript length, or presentation style? For example, this could include independent expert assessments or comparisons between videos matched on topic and length but differing in explanatory structure. Please also report per-level performance on an evaluation set containing enough independently selected examples at every CDS level, particularly levels 4–6. The current random gold set contains only two level-4 videos and no level-5 or level-6 videos, so the reported aggregate agreement provides limited evidence about the upper part of the scale.

2, Missingness and catalog composition: What fractions of each model’s top-K lists are null-labeled, and how do model rankings change under (i) null-as-zero, (ii) valid-only evaluation, (iii) multiple imputation or bounded sensitivity analysis, and (iv) candidate sets matched for category, popularity, and ASR availability? Please decompose the model–random gap relative to the candidate-pool depth distribution. This is important because 25.37% of all videos are null and 54.04% are level 0 (Fig. 5).

3, Statistical evidence: Across the five seeds, what are uncertainty intervals and paired tests/effect sizes for each LCDS difference versus the analytic random expectation and for the Recall–LCDS relationship? The paper reports means but does not establish which small differences in Table 6/11 are distinguishable from training variability.

4, Attribution: Please justify the statement that the preference “originates from the RSs themselves.” For the 44-user case study, what are the expected low-CDS shares under random, popularity-ranked, and category/popularity-matched baselines for each user’s candidate set? How sensitive is the result to user-selection and K?

5 comments
AuthorsOpenReview ↗

Response to W1: Validity and Reliability of CDS Evaluation

Thank you for your constructive comments. We have revised the manuscript accordingly and uploaded the updated version.

Critical Weaknesses

W1(1). Reliability shown, validity unproven.

In Appendix H (Surface-Level Correlates of CDS, p. 29), we analyze the correlation between the CDS assigned by human annotators and the surface-level attributes of videos, and we further conduct a robustness analysis.

VariableAssociationExplained variance
Log-Caption lengthSpearman ρ = 0.0241R² = 0.09%
Log-ASR lengthSpearman ρ = 0.1954R² = 5.93%
CategoryCramér V = 0.3217η² = 24.82%

As shown in the table above, the CDS assigned by human annotators is correlated with ASR length and category. Therefore, we conduct robustness tests on human annotators with respect to ASR length and category.

Category PerturbationP(Δs ≥ 1)
Low → Medium0.00%
Low → High0.00%
Medium → Low0.00%
High → Low0.00%
ASR PerturbationGroupδ|a|P(Δs = 1)P(Δs = 2)P(Δs > 2)
ASR length ↑Low+52.05%0.00%0.00%0.00%
ASR length ↑Medium+51.39%0.00%0.00%0.00%
ASR length ↓Medium−24.49%3.33%0.00%0.00%
ASR length ↓High−27.35%10.87%6.52%0.00%

As shown in the table above, neither replacing the category nor extending the ASR transcript changes CDS. Shortening the ASR transcript changes CDS only slightly, and all changes remain within two levels. These results indicate that human annotators assign CDS primarily based on video content rather than on shortcuts.

Furthermore, in our annotation workflow, each video is rated by three experts. When their ratings differ substantially, all five annotators rate the video to reach a consensus label. This procedure greatly reduces the risk that annotators rely on shortcuts or personal preferences.

W1(2) Higher levels under-tested.

We further expand the hard-sampled set from 50 to 170 samples, 3.4 times its original size. Together with the gold-standard subset, the expanded set contains over 3,000 human annotations covering 1,003 unique videos. Each CDS level has at least 50 samples, except level 6, which has only 6 samples in the entire dataset.

Human gold scorenSame score from Qwen3.7-Max
45774%
55290%
65100%

The table above reports the agreement between humans and Qwen3.7-Max on the new high-CDS samples of the hard-sampled set. Inter-annotator agreement is shown in Figure 10 (p. 25), and agreement with more LLMs is shown in Table 13 (p. 24). Overall, the expanded hard-sampled set covers all CDS levels. The results show that our rubric is operational and interpretable at every CDS level, further confirming that the scoring workflow aligns with human judgments.

W1(3). Need stronger validity checks of Human Evaluator.

We have addressed this concern together with W1(1) above.

AuthorsOpenReview ↗

Response to W2: Robustness, Statistical Significance, and Analysis of Low-CDS Recommendations

W2(1). Results confounded by nulls

In Appendix K (Handling of Null CDS Labels, p. 35), we analyze different imputation methods for null labels and compare their results with those under the valid-only setting. We consider global mean imputation, category mean imputation, multiple imputation, and constant imputation with values from 0 to 6. Constant imputation with values from 1 to 6 serves as a stress test, while the other methods are regarded as reasonable settings.

Imputation Strategy# in Low Region (/13)ΔRandom
Valid-only13−0.10
Global Mean13−0.09
Category Mean13+0.11
Multiple Imputation13+0.10
Constant 0 (Paper)13+0.12
Constant 113−0.32
Constant 213−0.76
Constant 313−1.20
Constant 411−1.64
Constant 53−2.08
Constant 61−2.52

The table above reports A-LCDS@20 on the Sampled dataset, and more results are shown in Figure 13 (p. 36). All RSs remain in the low-LCDS region under reasonable imputation methods. Even in the stress tests, they leave this region only when null labels are imputed with 5 or higher, which is unrealistic in practice. We also report Δ\DeltaRandom, defined as the mean result of the 13 RSs minus that of Random. Under all imputation methods, RSs stay close to Random, and the gap in favor of Random widens as the imputed value increases. These results confirm that both of our conclusions are robust: current RSs fall within the low-LCDS region and perform close to Random in terms of content depth.

W2(2). No significance testing.

DatasetMetricMedian95% CI range (13 RSs)Random
SampledA-LCDS@207.05[5.92, 8.02]6.89
SampledE-LCDS@207.16[5.34, 8.23]6.89
FullA-LCDS@206.97[6.50, 7.64]6.59
FullE-LCDS@207.10[6.43, 8.01]6.59

The table above reports the median and 95% Confidence Interval (CI) range of the 13 RSs on both datasets and metrics, and per-RS results are given in Table 12 (p. 22). The content depth performance of current RSs is close to that of Random baseline. These findings hold across five independent seeds, confirming that our conclusions are stable.

W2(3). Need further analysis of causality.

To address this concern, we have added an analysis on all users in the Further Analysis paragraph of Section 6 (SCOPE-Bench Leaderboard, pp. 9–10) in the revised manuscript. Specifically, we investigate why RSs exhibit a preference for low-CDS videos, which we refer to as OverRec for brevity, through the following four questions:

  • (a) Ranking. Is OverRec introduced by RS ranking?
  • (b) User Preference. Can OverRec be explained by an inherent preference of users for low-CDS videos?
  • (c) Category. Can OverRec be attributed to the difference in category between low-CDS and other videos?
  • (d) Popularity. Can OverRec be attributed to the difference in popularity between low-CDS and other videos?
(a) Ranking(b) User Preference(c) Category(d) Popularity
Median1.281.601.761.12
# RSs sig. > 1 (↑)1313131
# RSs sig. < 1 (↓)0000

The table above reports the results under the four settings, and detailed results for each RS are provided in Table 7 (p. 9). A ratio greater than 1 indicates that RSs favor low-CDS videos under the corresponding setting, and significance is determined by a Holm-corrected p<0.05p<0.05. The ratio is significantly greater than 1 for all RSs under settings (a), (b), and (c), whereas most RSs show no significant effect under setting (d). This result suggests that OverRec mainly stems from popularity bias: engagement-optimized RSs learn from collaborative signals and thus favor popular videos. Most popular videos have low CDS, since such videos appeal to a broader audience and capture attention more easily.

W2(4). Insufficient evidence to support the claim

First, we would like to clarify that we describe Recall and NDCG as measures of offline relevance, and we do not claim that they measure "attention". Furthermore, the Further Analysis paragraph of Section 6 (SCOPE-Bench Leaderboard, pp. 9-10) and our response to W2(3) provide a detailed analysis of why current RSs favor low-CDS videos. The main reason is that RSs optimized on collaborative engagement signals tend to recommend popular videos. Most of these videos have low content depth, because such videos appeal to a broader audience and capture attention more easily. Consequently, engagement-optimized RSs exhibit a preference for low-CDS videos.

AuthorsOpenReview ↗

Response to Q1–Q4: CDS Validation, Missingness Robustness, and Further Analysis of Recommendation Bias

Q1. Validate CDS, test all levels

This point is addressed together with W1. Please refer to our response to W1 above.

Q2. Missingness and catalog composition.

DatasetStatisticC@10C@20
SampledMean86.2486.68
FullMean74.6875.69

Regarding the coverage of valid CDS values, the table above reports the mean coverage over the 13 RSs at K=10K=10 and K=20K=20. More detailed coverage results are provided in Table 11 (p. 20).

MetricSampled SpearmanSampled RBOSampled PearsonFull SpearmanFull RBOFull Pearson
A-LCDS@100.970.940.990.880.900.95
A-LCDS@200.960.860.990.820.800.95
E-LCDS@100.990.950.990.890.880.94
E-LCDS@200.990.990.990.820.820.94

In addition, we compare the model rankings obtained with A-LCDS and E-LCDS against those obtained under the valid-only setting at K=10 and K=20. We measure ranking consistency using Spearman correlation, Rank-Biased Overlap (RBO), and Pearson correlation, and all three measures show very high consistency. This indicates that the model rankings obtained with zero imputation are reliable. The remaining comments are closely related to W2(1), and we have addressed them together. We kindly refer the reviewer to our response to W2(1) above.

Q3 Statistical evidence.

We have responded to this point jointly with W2(2). Please see our response to W2(2) above for details. In addition, to illustrate the relationship between Recall and LCDS more intuitively, the Training Trajectory paragraph and Figure 7 of Section 6 (SCOPE-Bench Leaderboard, p. 9) in the revised manuscript report the results of four RSs evaluated on the test set after each training epoch. As training progresses, the engagement metrics increase steadily, whereas the content-depth metrics remain largely flat and, in some cases, even decline. This observation further indicates that engagement optimization and content-depth optimization are decoupled in current RSs.

Q4. Attribution: Please justify the statement that the preference “originates from the RSs themselves.” For the 44-user case study, what are the expected low-CDS shares under random, popularity-ranked, and category/popularity-matched baselines for each user’s candidate set? How sensitive is the result to user-selection and K?

As this point overlaps with W2(3), we address it in our response to W2(3) above.

Reviewer_kK1mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a benchmark based on content depth. Motivated by the question, “Do recommender systems favor attention-grabbing content over videos with greater depth?”, it examines the relationship between attention and content depth and introduces the Content Depth Score to measure how deeply short videos present and convey information to viewers.

Critical strengths

S1: The evaluation perspective offers some novelty. Introducing content depth into short-video recommendation evaluation adds a content-side dimension beyond engagement. The annotation workflow is relatively complete: independent human annotation, disagreement resolution, automatic evaluator selection, and perturbation experiments on categories and transcript length provide some support for scalable scoring.

S2: Large-scale annotation and evaluation across multiple models provide a starting point for future research on content depth. The validation also has a reasonable foundation: human agreement analysis and bias-related perturbation experiments are more convincing than relying solely on automatic scoring.

Critical weaknesses

W1: There is a mismatch between the paper’s positioning and its empirical contribution. The paper is motivated by improving users’ cognition and well-being, but it evaluates content-depth measurement and recommendation distributions without establishing a link between CDS and user benefits. Thus, the evidence supports the claim that content depth can be assessed, but not why it matters. The scientific contribution implied by “Content Depth Matters” has therefore not yet been established: proposing a new evaluation dimension should not be equated with identifying a new recommendation objective that delivers value to users.

W2: The diagnostic conclusions are subject to confounding. Low scores may arise from the content supply and the handling of missing data; directly assigning zero to missing scores can also affect list-level evaluation. More importantly, a controlled user study is needed to better support the paper’s positioning. Such a study should match topic, duration, accuracy, and presentation quality as closely as possible, test whether content with different CDS levels leads to differences in actual comprehension, recall, and perceived depth, and examine the influence of users’ prior knowledge.

Questions for authors

Is CDS intended to measure content structure or predict user benefits? What evidence supports the latter interpretation? Do the conclusions about shallow-content preference hold after accounting for the candidate distribution and excluding unscorable items, rather than assigning them zero?

4 comments
AuthorsOpenReview ↗

Response to W1: Clarifying CDS and Its Relevance to Cognitive Outcomes

We appreciate your constructive feedback. We have uploaded a revised manuscript.

W1: Mismatch between what CDS measures and the claimed contributions

Content Depth Score (CDS) is a content-side metric. The CDS of a video quantifies the degree to which the video develops a topic from isolated information into structured understanding. In other words, CDS measures the extent to which a video provides opportunities for higher-order cognitive processing. It does not directly measure cognitive outcomes, just as a nutrition label describes the composition of food rather than its effects on health. However, this distinction does not imply that CDS is unrelated to the cognition and well-being of users. Rather, videos with higher CDS provide greater opportunities to improve the cognition and well-being of users.

Nevertheless, existing evidence already links content depth to cognitive outcomes. Compared with watching high-depth content, watching fast-paced, low-depth content immediately impairs the executive function of young children [1]. Both the levels-of-processing theory [2] and the ICAP framework [3] suggest that deeper content is more likely to elicit deeper cognitive processing, which in turn leads to better memory, understanding, and psychological well-being. In contrast, the consumption of fragmented, low-depth videos is associated with impaired prospective memory [4] and dissociative "watch-and-forget" experiences [5].

[1] Lillard et al. The immediate impact of different types of television on young children's executive function. Pediatrics, 2011.

[2] Craik et al. Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 1972.

[3] Chi et al. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 2014.

[4] Chiossi et al. Short-form videos degrade our capacity to retain intentions: Effect of context switching on prospective memory. CHI Conference on Human Factors in Computing Systems, 2023.

[5] Baughan et al. "I don't even remember what I read": How design influences dissociation on social media. CHI Conference on Human Factors in Computing Systems, 2022.

AuthorsOpenReview ↗

Response to W2: Robustness Analysis and User Study Validation

W2(1): The diagnostic conclusions are subject to confounding.

In Appendix K (Handling of Null CDS Labels, p. 35), we add an analysis of different imputation methods for null labels and compute the consistency between the results obtained with each method and those obtained under the Valid Only setting. Specifically, the imputation methods include global mean imputation, category mean imputation, multiple imputation, and constant imputation with values of 0, 1, 2, 3, 4, 5, and 6. Among them, constant imputation with values from 1 to 6 serves as a stress test, whereas the remaining methods serve as reasonable imputation settings.

Imputation StrategySampled @5Sampled @10Sampled @20
Valid-only (Ref.)–––
Global Mean1.000.991.00
Category Mean0.961.000.95
Multiple Imputation0.960.990.93
Constant 0 (Paper)0.980.970.96
Constant 10.970.990.95
Constant 20.910.930.90
Constant 30.870.900.82
Constant 40.770.790.76
Constant 50.660.740.69
Constant 60.600.690.68

The table above reports the Spearman correlation coefficients for A-LCDS. The results for E-LCDS, additional consistency metrics, and the full dataset are provided in Table 23 (p. 37). As shown in these results, global mean imputation, category mean imputation, multiple imputation, and constant imputation with 0 all yield model rankings that are highly consistent with those obtained under the valid-only setting. In the main text, we adopt constant imputation with 0 primarily because it provides a lower bound.

Imputation Strategy# in Low Region (/13)ΔRandom
Valid-only13−0.10
Global Mean13−0.09
Category Mean13+0.11
Multiple Imputation13+0.10
Constant 0 (Paper)13+0.12
Constant 113−0.32
Constant 213−0.76
Constant 313−1.20
Constant 411−1.64
Constant 53−2.08
Constant 61−2.52

The table above reports the results of A-LCDS@20 on the Sampled dataset, and more results are provided in Figure 13 (p. 36). As shown in these results, all RSs remain in the low-LCDS region under reasonable imputation methods. Even under the stress tests, current RSs move out of this region only when null labels are imputed with a constant value of 5 or higher, which is unrealistic in practice. In addition, we report Δ\DeltaRandom, defined as the mean result of the 13 RSs minus the result of Random. Regardless of the imputation method, the performance of RSs remains close to that of Random, and the margin by which Random outperforms RSs even widens as the imputed value increases. Therefore, these results demonstrate that both of our conclusions are robust: current RSs fall within the low-LCDS region, and their performance on content-depth metrics is close to that of Random.

W2(2): A controlled user study.

To examine whether CDS reflects the benefit of watching a video, we conduct a user study. We construct five groups of videos. Each group contains one low-, one medium-, and one high-CDS video on the same category and knowledge point. The ASR transcripts within each group differ in length by less than 10%. Five participants who have not watched any of these videos before take part in the study. Each participant independently watches all five groups. Within each group, the videos are shown in random order. The annotator then ranks them by perceived benefit.

This procedure yields 25 rankings, all of which follow the order high >> medium >> low. These results suggest that CDS, although defined at the content level, also reflects the benefit of watching a video. This finding is consistent with prior studies [1,2] and with our central claim that deeper content is more likely to support deeper cognitive processing.

[1] Craik et al. Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 1972.

[2] Chi et al. The ICAP framework: Linking cognitive engagement to active learning outcomes. Educational Psychologist, 2014.

AuthorsOpenReview ↗

Follow-up: summary of our responses

Thank you again for your review. With the discussion period nearing its end, we briefly summarize how our rebuttal addresses your concerns and where the supporting evidence appears in the revised manuscript (new text in blue).

  • W1 (what CDS measures): CDS is a content-side measure of how far a video develops a topic into structured understanding. It describes the opportunity for deeper cognitive processing that a video offers, not the cognitive outcome itself, much as a nutrition label describes the composition of food rather than its effect on health. Prior work in cognitive science and HCI, cited in our rebuttal, links content depth to cognitive outcomes such as memory, understanding, and well-being.
  • W2(1) (confounding): App. K (pp. 35–37) compares valid-only evaluation with global-mean, category-mean, multiple and constant (0–6) imputation. Under every reasonable strategy, model rankings stay highly consistent with the valid-only setting (Spearman 0.93–1.00 for A-LCDS on Sampled, Table 23), and all 13 RSs remain in the Low-LCDS region (Fig. 13).
  • W2(2) (user study): In our controlled study, five participants each ranked five matched groups. Each group contains one low-, one medium- and one high-CDS video on the same knowledge point, with transcript lengths differing by less than 10%. All 25 rankings place perceived benefit in the order high > medium > low. This result agrees with the levels-of-processing theory and the ICAP framework cited in our rebuttal, which hold that deeper content elicits deeper cognitive processing and in turn better memory and understanding.

Together, these results show that CDS is a content-side measure consistent with both prior evidence and users' perceived benefit, and that our diagnostic conclusions are robust to null handling.

We would appreciate your taking a look at our rebuttal. If it resolves your concerns, please consider revising your score upward. We would be happy to clarify any remaining questions.

AuthorsOpenReview ↗

Gentle Reminder: Discussion Period Ends in 12 Hours

Dear Reviewer kK1m,

We apologize for following up again. With less than 12 hours left in the discussion period, we would like to check whether our responses have addressed your concerns:

  • W1 (what CDS measures): CDS is a content-side measure of the opportunity for deeper cognitive processing. Its link to user benefit is supported by prior work in cognitive science and HCI.
  • W2 (confounding and user study): App. K shows that our conclusions hold under valid-only evaluation and alternative null-imputation strategies. A controlled user study shows that perceived benefit follows the order high > medium > low CDS.

If any concern remains, please let us know and we will respond promptly. If our responses have resolved your concerns, we would sincerely appreciate it if you could consider updating your score.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank Reviewers kK1m, ygpZ, and zpFU for their constructive reviews. We have uploaded a revised manuscript in which new and revised text is marked in blue.

All three reviewers recognize the same core strengths:

  1. A new evaluation axis. SCOPE-Bench brings content depth into short-video recommendation evaluation. Engagement metrics do not capture this content-side dimension (kK1m, ygpZ, zpFU).
  2. A clear and inspectable framework. The seven-level CDS rubric, the LCDS metrics, the prompt, the boundary rules, and the null-label handling are fully documented (ygpZ). The presentation is clear and easy to follow (zpFU).
  3. Human-aligned and robust scoring. The LLM evaluator produces CDS labels close to human annotations. Human agreement analysis, a hard-sampled set, and category and verbosity perturbations support this result (kK1m, ygpZ, zpFU).
  4. A resource for future work. SCOPE-bench give a starting point for content-depth-aware recommendation (kK1m, zpFU).

The shared concerns are about what CDS measures, how null labels affect the conclusions, and why current RSs favor low-depth videos. We summarize our responses below. The per-reviewer replies give full tables.

1. What CDS measures (kK1m-W1, W2)

CDS is a content-side measure of how far a video develops a topic into structured understanding. It does not directly measure cognitive outcomes. Our contribution is to make content depth measurable at scale. Prior work in cognitive science and a controlled user study, in which all 25 rankings place perceived benefit as high > medium > low CDS, support its link to user benefit.

2. Validity of CDS (ygpZ-W1, Q1)

Counterfactual tests show that human labels are not driven by category or transcript length (App. H, pp. 29--30). The hard-sampled set has been expanded from 50 to 170 videos and now covers every CDS level. Combined with the gold-standard subset, each level except level 6 contains 50+ videos. Moreover, Qwen3.7-Max remains aligned with human annotators at levels 4--6 (App. D.3, pp. 24--26).

3. Null labels (kK1m-W2, ygpZ-W2, Q2)

Under valid-only evaluation and every reasonable imputation strategy, model rankings stay consistent and all 13 RSs remain in the Low-LCDS region (App. K, pp. 35–37).

4. Statistics across five seeds (ygpZ-W2, Q3)

The five-seed means and 95% CIs of all RSs stay in the Low region and close to Random (Table 12, p. 22).

5. Engagement vs. content depth (ygpZ-W2, Q3)

As training proceeds, Recall and NDCG rise steadily while A-LCDS and E-LCDS stay flat or decline (Fig. 7, p. 9). Engagement gains therefore do not bring deeper content.

6. Why RSs favor low-CDS videos (ygpZ-W2, Q4)

We add an analysis of RSs on all users. The low-CDS preference persists after controlling for ranking, user preference, and category, but largely disappears once popularity is matched (Sec. 6, pp. 9--10; App. J, pp. 33--35). This preference therefore mainly reflects popularity bias. Engagement-optimized RSs are trained on collaborative signals and thus favor popular videos. Meanwhile, popular videos are dominated by low-CDS videos, which appeal to a broader audience and capture attention more easily. As a result, popularity bias leads current RSs to favor low-CDS videos.

Summary

In short, CDS is a validated content-side measure that is not driven by surface features, and the finding that current RSs recommend low-depth content holds across null-handling strategies and seeds. Engagement gains do not translate into content depth, and the low-depth preference is mainly explained by popularity bias.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.