- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper focuses on estimating the true marginal value of items in recommender ecosystems and proposes SCSV, which models substitution and compensation effects through a shielding estimand and validates the method with large-scale randomized shielding experiments and online A/B tests. However, SCSV is mainly composed of existing techniques and heuristic mechanisms, resulting in limited algorithmic novelty. Its core compensation correction relies on handcrafted features and calibration using historical randomized experiments, while the current validation is largely conducted at the set level and is therefore insufficient to demonstrate that item-level credit can accurately characterize the true marginal contribution of individual items. In addition, the paper lacks systematic ablation studies of key components, and the dataset and data-splitting details are not sufficiently complete, which limits the assessment of the method’s effectiveness and reproducibility.
Critical strengths
-
The authors propose a new method that measures item value through a shielding estimand.
-
The authors construct an industrial-scale evaluation benchmark.
Critical weaknesses
-
The proposed method is primarily a combination of heuristic rules inspired by existing techniques and does not provide substantial algorithmic novelty. Overall, the paper appears highly engineering-oriented and reads more like a technical report than an academic research paper. For example, the proposed SCSV method essentially combines Lagged Differencing, Double Machine Learning (DML), Replaceability Discount, Sign Gating, and Credit Settlement to estimate the marginal ecosystem value of an item. While these designs have certain engineering rationality, the paper does not sufficiently explain why this particular combination of mechanisms is necessary, nor does it demonstrate that the proposed combination has an irreplaceable advantage over alternative credit assignment or causal correction methods. In particular, the most critical component of the paper, Substitution Correction, actually relies on a clipped and calibrated compensation ratio rather than directly estimating the true counterfactual substitution effect. Therefore, it remains unclear whether SCSV constitutes a genuinely new causal value estimation method or is essentially a heuristic post-processing procedure applied to existing causal attribution results.
-
The core objective of the paper is to estimate the true marginal contribution of an item to the recommender ecosystem. However, the proposed SCSV Credit is not a direct estimate of the true counterfactual effect, but rather an operational credit score constructed through DML Attribution, Sign Gating, Replaceability Discount, and Credit Settlement. In particular, the key Compensation Ratio relies on substitute-density features and calibration using historical randomized experiments, while its effectiveness under changes in the supply structure or user preferences is not sufficiently validated. Moreover, clipping the Compensation Ratio to may assign zero credit to items with genuinely negative marginal value, potentially distorting the value estimation. The authors also explicitly acknowledge that SCSV Credit is not a consistent estimator of the true item-level Shielding Effect. Although the paper validates the accuracy of aggregated estimates through set-level randomized shielding experiments, agreement at the set level does not guarantee accurate credit assignment at the individual-item level, since estimation errors across different items may cancel each other out after aggregation. Therefore, the current theoretical and experimental evidence is still insufficient to fully support the central claim that SCSV can accurately characterize the true marginal ecosystem contribution of individual items.
-
Although the paper provides large-scale randomized shielding experiments and online A/B experiments, it lacks systematic ablation studies on the core algorithmic components, making it difficult to identify the actual source of the performance improvements. SCSV combines multiple mechanisms, including Lagged Differencing, DML Attribution, Sign Gating, Replaceability Discount, and Credit Settlement, while the current experiments mainly focus on overall performance comparisons and do not sufficiently validate the independent contribution and necessity of each component. In particular, the authors should further compare the performance after removing Sign Gating, Replaceability Discount, or Credit Settlement, in order to clarify the actual contribution of Substitution Correction to the final performance gains.
-
The authors claim to contribute an industrial-scale evaluation dataset, but important information about the dataset is missing. For example, the paper does not clearly report the number of users, the number of items, the scale of interactions, the exact numbers of training and test samples, or the time span covered by the data. In addition, the authors only state that a temporal split is adopted, without sufficiently specifying the exact splitting strategy, data filtering rules, or the data selection criteria used for different experimental scenarios. As a result, it is difficult to assess the appropriateness of the experimental setup and the potential risk of data leakage.
Questions for authors
Please see above weakness.
10 comments
W1 (2/2): Methodological Contribution and the Calibrated Substitution Correction
Role of the settlement steps. Deconfounding alone does not specify how item-level attribution should be settled under substitution. The gate admits contributions whose direction agrees with the unit's realized change, the discount removes the share that the remaining supply is estimated to recover, and the unit budget bounds the credit a unit settles. For the chosen discounted masses and unit budget, the settlement magnitudes are the generalized KL projection onto the budget set, which never raises a mass and preserves the within-unit order (Proposition 2.1, Appendix C). In the leave-one-out ablation, removing any one of the three steps increases the deviation from the randomized effect on all four fixed sets and at all five shielding ratios. With the discount removed, the estimates are about 2 to 13 times the randomized loss on the four fixed sets (Table 14, Appendix J.1).
Comparison with alternatives. The baseline implementations in Table 2 (IPS, propensity-weighted value, and doubly robust) use the same logs, covariates, and folds. Each fits one pooled slope per unit of exposure share, so it returns a set prediction that is linear in the removed share and no item-level credit (Appendix G.1). SCSV has the smallest deviation and standard error, and the doubly robust estimate is also close in aggregate. SCSV additionally uses the earlier randomized rounds through its compensation model, so Table 2 compares complete methods and does not isolate the settlement structure. For supply selection, 13% of supply ranked by the compensation-aware score loses more when shielded (−2.167%) than 22% ranked by an uncompensated score (−1.924%; Appendix J), and a prior trained on settled credits raises open-reason ROI from 13.99% to 50.00% in production (Table 4).
Revision. The Introduction now lists three numbered contributions. Section 2 opens with an overview of the four steps and of how randomized shielding and online experiments test them. Appendix J.1 and Section 3.2 report the ablation, and Appendix M.2 compares SCSV with the structural alternative (Table 16). The changes are highlighted in blue.
We hope this response has addressed your concerns.
W2 (1/2): Item-Level Validity of the Settled Credit
Thank you for your detailed comments on the validity and interpretation of the settled credits. We answer the four points in turn.
Item-level consistency. As Section 2.1 states, SCSV is an operational credit construction and is not claimed to be a consistent estimator of each item's shielding effect. At the full-platform scale described in Appendix H, individual-item effects on platform-level metrics are generally difficult to detect, so randomized shielding is run on sets, and every comparison with a randomized effect is made at that level (Section 3). The settled credit is therefore tested in the two respects that supply intervention relies on: its value, against randomized effects on sets, and its ordering, through the supply selected by it.
Set-level prediction, supply selection, and online intervention. Our experiments evaluate three complementary uses of SCSV's item-level credits: predicting losses on held-out sets, selecting supply with high system-level value, and guiding online intervention.
Prediction on held-out interventions. A shared compensation parameter vector is calibrated on 10 earlier randomized shielding rounds and frozen before the reported validation experiments. Predictions are compared with randomized effects on sets of different composition: a random 20% of core authors (−1.350% predicted vs −1.351% observed), the top 10% of items by long-term retention value (−0.95% vs −1.25%), and 5% of the highest streamer tier on live revenue (−1.64% vs −1.83%). The estimates also track the sign, ordering, and qualitative convex profile of losses across five watch-time shielding doses from 10% to 90% (Table 1), and the promotion comparison gives +0.21% predicted versus +0.230% observed. These comparisons test transfer of the historical calibration to new intervention sets, metrics, doses, and directions (Section 3, Appendix J).
Supply selection. Shielding 13% of supply selected by the compensation-aware score reduces open reasons by 2.167%, compared with 1.924% for 22% selected by an uncompensated score (Appendix J). In this comparison, a smaller selected set carries a larger system-level impact, providing experimental evidence for the credit's value in supply selection.
Online intervention. In the user-randomized online experiment, the policy guided by a prior trained on settled credits increases open reasons by 0.363% (). The reported open-reason intervention ROI reaches 50.00%, compared with 13.99% for the incumbent strategy and approximately 19% for random boosting (Table 4, Appendix L). This evaluates the value signal through the system-level gains achieved when it is used for exposure decisions.
Together, these results support SCSV's aggregate predictive and supply-intervention value. Consistent with Section 2.1, these conclusions concern the evaluated sets and policies; they do not imply that every item's credit has been independently validated as its causal effect.
W1 (1/2): Methodological Contribution and the Calibrated Substitution Correction
Thank you for your careful assessment and for asking us to clarify the methodological contribution.
Our methodological contribution is a credit construction that depends on the supply remaining after a removal and is calibrated on randomized set removals. It combines item-level attribution from production logs with a shared compensation model, so that credits adapt to the substitutes available under each removal scenario. This yields predictions for held-out removal sets and doses without a separate experiment for every item, and the resulting credits are the training labels for upload-time supply selection. Lagged differencing and DML are standard tools that supply the attribution inputs, and any orthogonalized estimator can replace the DML step (Appendix B).
Why the substitution correction is calibrated. In the data used here, logs do not realize the post-removal reallocation of an arbitrary item. Randomized shielding is run on sets and identifies their net loss. Neither arm removes supply without reallocation, so the gross loss of the removed supply and the utility that substitutes recover are not identified separately (Appendix B.1), and this split requires a model. In the response model, the lagged response of a shielded set equals its gross loss times 1−r (Eq. (17)), and the same identity holds to first order for an item (Appendix F.2). This decomposition motivates the form of the discount. The production ratio is a shared parametric approximation. It is a clipped linear function of substitute-density features of the remaining supply, with one parameter vector shared across items, periods, and metrics, calibrated once against the set-level net losses of 10 earlier shielding rounds and frozen (Appendix J). The clipping to [0,1] is a stated restriction, which our response to point 2 discusses. A structural demand model reaches the net value by simulating removals from a fitted demand system (Zielnicki et al., 2025). SCSV does not fit a demand system, and it calibrates on removals in which the production ranker performs the reallocation (Appendix M.2).
What the construction enables. With the compensation-model parameters frozen, the settled estimates are compared with randomized experiments outside the calibration rounds that differ in cohort, metric, dose, and direction: a random 20% of core authors (−1.350% against −1.351%), the top 10% of items by long-term retention value, five shielding ratios, live revenue, and a promotion. Every deviation is reported and is at most 0.67 pp, including a consistent shortfall on watch time (Section 3). The full SCSV estimates reproduce the observed sign, ordering, and qualitative convex dose profile (Table 1). By construction, a ratio held fixed would make the estimate linear in the shielded fraction under random shielding (Appendix J).