acceptodds
Under review as a conference paper at ICLR 2027

SCSV: Substitution-Corrected Supply Valuation for Value-Guided Intervention in Recommender Ecosystems

Abstract

Recommender platforms continuously invest attention in new items and creators, yet under finite attention the engagement an item attracts may be reallocated from substitutes rather than created. Engagement-based estimators overlook this substitution, while randomized shielding is difficult to scale to item-level valuation. We propose Substitution-Corrected Supply Value (SCSV), which values an item by a shielding estimand, the ecosystem utility lost when it is removed and attention reallocates to substitutes. This value can be settled from production logs, agrees with randomized shielding at shielded-set level, and improves positive supply intervention in production. SCSV uses lagged differencing against exposure leakage and double machine learning against observed confounding. Its settlement is sign-gated and replaceability-discounted, and caps each attribution unit's credit by its realized utility change. The intervention layer distills settled credits into uncertainty-aware priors, ranks candidates by risk-adjusted value density, and allocates exposure only when gains exceed displaced value, with conditional bounds on selection optimality and allocation error. We construct the first industrial-scale benchmark pairing production logs with randomized supply-shielding ground truth. Set-level estimates agree with these shielding losses in sign and ordering, within 0.3 pp on user-initiated content entries (open reasons) and with dose-dependent gaps on watch time. SCSV identifies supply with stronger system-level impact than incumbent metrics, improving selection efficiency by 226%, 90.3%, and 46.9% across cold-start, production, and live-streaming scenarios. In online experiments with at least 20 million daily active users, SCSV improves watch time, open reasons, and long-term retention, while raising open-reason intervention ROI to 3.6 times the baseline. These results support substitution-corrected valuation for directing supply investment toward utility that is created rather than reallocated.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Substitution utility, Supply valuation, Recommender ecosystems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gwr7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on estimating the true marginal value of items in recommender ecosystems and proposes SCSV, which models substitution and compensation effects through a shielding estimand and validates the method with large-scale randomized shielding experiments and online A/B tests. However, SCSV is mainly composed of existing techniques and heuristic mechanisms, resulting in limited algorithmic novelty. Its core compensation correction relies on handcrafted features and calibration using historical randomized experiments, while the current validation is largely conducted at the set level and is therefore insufficient to demonstrate that item-level credit can accurately characterize the true marginal contribution of individual items. In addition, the paper lacks systematic ablation studies of key components, and the dataset and data-splitting details are not sufficiently complete, which limits the assessment of the method’s effectiveness and reproducibility.

Critical strengths
  1. The authors propose a new method that measures item value through a shielding estimand.

  2. The authors construct an industrial-scale evaluation benchmark.

Critical weaknesses
  1. The proposed method is primarily a combination of heuristic rules inspired by existing techniques and does not provide substantial algorithmic novelty. Overall, the paper appears highly engineering-oriented and reads more like a technical report than an academic research paper. For example, the proposed SCSV method essentially combines Lagged Differencing, Double Machine Learning (DML), Replaceability Discount, Sign Gating, and Credit Settlement to estimate the marginal ecosystem value of an item. While these designs have certain engineering rationality, the paper does not sufficiently explain why this particular combination of mechanisms is necessary, nor does it demonstrate that the proposed combination has an irreplaceable advantage over alternative credit assignment or causal correction methods. In particular, the most critical component of the paper, Substitution Correction, actually relies on a clipped and calibrated compensation ratio rather than directly estimating the true counterfactual substitution effect. Therefore, it remains unclear whether SCSV constitutes a genuinely new causal value estimation method or is essentially a heuristic post-processing procedure applied to existing causal attribution results.

  2. The core objective of the paper is to estimate the true marginal contribution of an item to the recommender ecosystem. However, the proposed SCSV Credit is not a direct estimate of the true counterfactual effect, but rather an operational credit score constructed through DML Attribution, Sign Gating, Replaceability Discount, and Credit Settlement. In particular, the key Compensation Ratio relies on substitute-density features and calibration using historical randomized experiments, while its effectiveness under changes in the supply structure or user preferences is not sufficiently validated. Moreover, clipping the Compensation Ratio to [0,1][0,1] may assign zero credit to items with genuinely negative marginal value, potentially distorting the value estimation. The authors also explicitly acknowledge that SCSV Credit is not a consistent estimator of the true item-level Shielding Effect. Although the paper validates the accuracy of aggregated estimates through set-level randomized shielding experiments, agreement at the set level does not guarantee accurate credit assignment at the individual-item level, since estimation errors across different items may cancel each other out after aggregation. Therefore, the current theoretical and experimental evidence is still insufficient to fully support the central claim that SCSV can accurately characterize the true marginal ecosystem contribution of individual items.

  3. Although the paper provides large-scale randomized shielding experiments and online A/B experiments, it lacks systematic ablation studies on the core algorithmic components, making it difficult to identify the actual source of the performance improvements. SCSV combines multiple mechanisms, including Lagged Differencing, DML Attribution, Sign Gating, Replaceability Discount, and Credit Settlement, while the current experiments mainly focus on overall performance comparisons and do not sufficiently validate the independent contribution and necessity of each component. In particular, the authors should further compare the performance after removing Sign Gating, Replaceability Discount, or Credit Settlement, in order to clarify the actual contribution of Substitution Correction to the final performance gains.

  4. The authors claim to contribute an industrial-scale evaluation dataset, but important information about the dataset is missing. For example, the paper does not clearly report the number of users, the number of items, the scale of interactions, the exact numbers of training and test samples, or the time span covered by the data. In addition, the authors only state that a temporal split is adopted, without sufficiently specifying the exact splitting strategy, data filtering rules, or the data selection criteria used for different experimental scenarios. As a result, it is difficult to assess the appropriateness of the experimental setup and the potential risk of data leakage.

Questions for authors

Please see above weakness.

10 comments
AuthorsOpenReview ↗

W1 (1/2): Methodological Contribution and the Calibrated Substitution Correction

Thank you for your careful assessment and for asking us to clarify the methodological contribution.

Our methodological contribution is a credit construction that depends on the supply remaining after a removal and is calibrated on randomized set removals. It combines item-level attribution from production logs with a shared compensation model, so that credits adapt to the substitutes available under each removal scenario. This yields predictions for held-out removal sets and doses without a separate experiment for every item, and the resulting credits are the training labels for upload-time supply selection. Lagged differencing and DML are standard tools that supply the attribution inputs, and any orthogonalized estimator can replace the DML step (Appendix B).

Why the substitution correction is calibrated. In the data used here, logs do not realize the post-removal reallocation of an arbitrary item. Randomized shielding is run on sets and identifies their net loss. Neither arm removes supply without reallocation, so the gross loss of the removed supply and the utility that substitutes recover are not identified separately (Appendix B.1), and this split requires a model. In the response model, the lagged response of a shielded set equals its gross loss times 1−r1-r (Eq. (17)), and the same identity holds to first order for an item (Appendix F.2). This decomposition motivates the form of the discount. The production ratio is a shared parametric approximation. It is a clipped linear function of substitute-density features of the remaining supply, with one parameter vector shared across items, periods, and metrics, calibrated once against the set-level net losses of 10 earlier shielding rounds and frozen (Appendix J). The clipping to [0,1][0,1] is a stated restriction, which our response to point 2 discusses. A structural demand model reaches the net value by simulating removals from a fitted demand system (Zielnicki et al., 2025). SCSV does not fit a demand system, and it calibrates on removals in which the production ranker performs the reallocation (Appendix M.2).

What the construction enables. With the compensation-model parameters frozen, the settled estimates are compared with randomized experiments outside the calibration rounds that differ in cohort, metric, dose, and direction: a random 20% of core authors (−1.350% against −1.351%), the top 10% of items by long-term retention value, five shielding ratios, live revenue, and a promotion. Every deviation is reported and is at most 0.67 pp, including a consistent shortfall on watch time (Section 3). The full SCSV estimates reproduce the observed sign, ordering, and qualitative convex dose profile (Table 1). By construction, a ratio held fixed would make the estimate linear in the shielded fraction under random shielding (Appendix J).

AuthorsOpenReview ↗

W1 (2/2): Methodological Contribution and the Calibrated Substitution Correction

Role of the settlement steps. Deconfounding alone does not specify how item-level attribution should be settled under substitution. The gate admits contributions whose direction agrees with the unit's realized change, the discount removes the share that the remaining supply is estimated to recover, and the unit budget bounds the credit a unit settles. For the chosen discounted masses and unit budget, the settlement magnitudes are the generalized KL projection onto the budget set, which never raises a mass and preserves the within-unit order (Proposition 2.1, Appendix C). In the leave-one-out ablation, removing any one of the three steps increases the deviation from the randomized effect on all four fixed sets and at all five shielding ratios. With the discount removed, the estimates are about 2 to 13 times the randomized loss on the four fixed sets (Table 14, Appendix J.1).

Comparison with alternatives. The baseline implementations in Table 2 (IPS, propensity-weighted value, and doubly robust) use the same logs, covariates, and folds. Each fits one pooled slope per unit of exposure share, so it returns a set prediction that is linear in the removed share and no item-level credit (Appendix G.1). SCSV has the smallest deviation and standard error, and the doubly robust estimate is also close in aggregate. SCSV additionally uses the earlier randomized rounds through its compensation model, so Table 2 compares complete methods and does not isolate the settlement structure. For supply selection, 13% of supply ranked by the compensation-aware score loses more when shielded (−2.167%) than 22% ranked by an uncompensated score (−1.924%; Appendix J), and a prior trained on settled credits raises open-reason ROI from 13.99% to 50.00% in production (Table 4).

Revision. The Introduction now lists three numbered contributions. Section 2 opens with an overview of the four steps and of how randomized shielding and online experiments test them. Appendix J.1 and Section 3.2 report the ablation, and Appendix M.2 compares SCSV with the structural alternative (Table 16). The changes are highlighted in blue.

We hope this response has addressed your concerns.

AuthorsOpenReview ↗

W2 (1/2): Item-Level Validity of the Settled Credit

Thank you for your detailed comments on the validity and interpretation of the settled credits. We answer the four points in turn.

Item-level consistency. As Section 2.1 states, SCSV is an operational credit construction and is not claimed to be a consistent estimator of each item's shielding effect. At the full-platform scale described in Appendix H, individual-item effects on platform-level metrics are generally difficult to detect, so randomized shielding is run on sets, and every comparison with a randomized effect is made at that level (Section 3). The settled credit is therefore tested in the two respects that supply intervention relies on: its value, against randomized effects on sets, and its ordering, through the supply selected by it.

Set-level prediction, supply selection, and online intervention. Our experiments evaluate three complementary uses of SCSV's item-level credits: predicting losses on held-out sets, selecting supply with high system-level value, and guiding online intervention.

Prediction on held-out interventions. A shared compensation parameter vector is calibrated on 10 earlier randomized shielding rounds and frozen before the reported validation experiments. Predictions are compared with randomized effects on sets of different composition: a random 20% of core authors (−1.350% predicted vs −1.351% observed), the top 10% of items by long-term retention value (−0.95% vs −1.25%), and 5% of the highest streamer tier on live revenue (−1.64% vs −1.83%). The estimates also track the sign, ordering, and qualitative convex profile of losses across five watch-time shielding doses from 10% to 90% (Table 1), and the promotion comparison gives +0.21% predicted versus +0.230% observed. These comparisons test transfer of the historical calibration to new intervention sets, metrics, doses, and directions (Section 3, Appendix J).

Supply selection. Shielding 13% of supply selected by the compensation-aware score reduces open reasons by 2.167%, compared with 1.924% for 22% selected by an uncompensated score (Appendix J). In this comparison, a smaller selected set carries a larger system-level impact, providing experimental evidence for the credit's value in supply selection.

Online intervention. In the user-randomized online experiment, the policy guided by a prior trained on settled credits increases open reasons by 0.363% (p=0.019p=0.019). The reported open-reason intervention ROI reaches 50.00%, compared with 13.99% for the incumbent strategy and approximately 19% for random boosting (Table 4, Appendix L). This evaluates the value signal through the system-level gains achieved when it is used for exposure decisions.

Together, these results support SCSV's aggregate predictive and supply-intervention value. Consistent with Section 2.1, these conclusions concern the evaluated sets and policies; they do not imply that every item's credit has been independently validated as its causal effect.

Reviewer_wLrHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper highlights that attracting engagement to an item does not necessarily increase overall platform utility, since it may simply take engagement from other items. To address this, it proposes SCSV (Substitution-Corrected Supply Value). It constructs item-level credits from production logs and uses a learned predictor of these credits to guide exposure for new items.

Critical strengths

a) The paper addresses a practically important and interesting problem. Attracting engagement does not necessarily translate into increased overall platform utility. b) The evaluation includes randomized shielding experiments and large-scale online deployment, with improvements in several platform metrics.

Critical weaknesses

a) The writing makes the paper unnecessarily difficult to follow. While the main problem is understandable, the introduction quickly shifts into technical terminology and lists of components without adequately explaining their connection. For example, lines 79–91 introduce terms like attribution, settlement, and allocation mechanisms with little intuition. The methodology similarly interrupts basic definitions and has appendix references before explaining the complete procedure. This leaves to the reader to reconstruct how the method works. The paper needs substantial revision to improve its flow and accessibility. b) The novelty needs clearer positioning. For example, The Value of Personalized Recommendations: Evidence from Netflix (Zielnicki et al.) already discusses item- and set-level removal value while accounting for substitution. The authors should clarify what their credit construction and intervention mechanism add beyond this existing valuation objective. c) The method does not clearly justify how each item receives its credit. It allows different attribution weights to explain the same observed changes while giving different credits to individual items. Since these credits are important in determining which items receive more exposure, the paper should explain how this ambiguity is resolved and whether the choice affects credit signs or item rankings. d) The ablations do not sufficiently isolate the individual design choices. They do not separately establish the contributions of parts such as sign gate, replaceability discount, and credit cap. It therefore remains unclear whether all these steps are necessary and how much each contributes to the reported improvements.

Questions for authors

Please refer to the weaknesses.

8 comments
AuthorsOpenReview ↗

W1: Connections between Components and the Flow of Section 2

Thank you for your careful reading and specific suggestions for improving the clarity of the Introduction and methodology. We address the two parts in turn.

Connections in lines 79–91. Lines 79–91 summarize our contributions, so several technical terms appear there in compact form before they are defined, and we apologize for the difficulty this created. Each of these terms is defined and explained in Section 2, and the problems the components address are introduced earlier in the Introduction (lines 49–72). All components serve one target, the shielding estimand: the next-period ecosystem utility lost when an item is removed and its attention is reallocated (lines 49–50; Sec. 2.1, Eq. (1)). The table below summarizes how each group of components addresses these problems and feeds the next step.

GroupChallenge (Introduction)Role and connection to the next stepDefined in
Attribution: lagged differences, orthogonalization (DML)Ranking decisions shape exposure and responses feed back into future exposure, so attribution faces exposure leakage and high-dimensional confounding (lines 51–53)Relates each unit's earlier change in item-level behavior to its later utility change, so the response is measured after the exposure change, and partials out observed confounders. The result is a signed predicted contribution for each item, the input to settlement.Sec. 2.2, Eq. (3)
Settlement: sign gate, replaceability discount, unit budgetA deconfounded attribution still credits utility that substitutes would recover, and credits summed within a user need not respect that user's realized change (lines 53–55, 64–66)Admits only contributions aligned with the unit's observed utility change, since a conflicting contribution may reflect a substitute's compensation or noise rather than the item's own effect; discounts the share that substitutes could recover; and bounds the unit's total absolute credit by the magnitude of its observed utility change. The resulting credits are the labels for the next step.Sec. 2.1; Sec. 2.2, Eqs. (4)–(6); App. A
Allocation: uncertainty-aware prior, density-ordered selection, displacement-aware allocationNew supply must be selected before any credit is settled, from predictions whose reliability varies; support must weigh exposure costs and the value displaced from substitutes (lines 66–72; Fig. 1)A prior trained on settled credits predicts the credit and its uncertainty at upload (lines 87–88). Selection ranks candidates by predicted value per unit of exposure cost, and allocation scores each action net of the value of the supply it displaces. The deployed policy gates new items on predicted value and uncertainty and passes eligible items to the existing ranker.Sec. 2.3; Sec. 4

These connections are now also stated in the paper, in a new overview at the start of Section 2.

Flow of Section 2. Following your comments on the methodology, we have revised Section 2 so that the complete procedure is described before the technical details. All changes are highlighted in blue, and the equation numbering in Section 2 is unchanged.

  • Overview of the full procedure. Section 2 now opens with an overview of the four steps (attribution, settlement, upload-time prediction, and exposure support), each introduced by the problem it addresses, followed by how randomized shielding and online experiments evaluate them. All appendix references in Section 2 appear after this overview.
  • Uninterrupted definitions. The shielding estimand is defined in one passage, from its target to Eq. (1). The discussion of the next-period window and of exposure conservation follows the definition in a separate paragraph, "Scope of the estimand", and the technical link to the shielding experiments is kept in Appendix B.1.
  • Key quantities in place. The reference score σ\sigma that orients credit is displayed where it is first used (Sec. 2.2), and the risk-adjusted score Pi,t=[πi,t−κui,t]+P_{i,t}=[\pi_{i,t}-\kappa u_{i,t}]_+ is defined in Sec. 2.3 together with its symbols.
  • Appendix references after explanations. References within definitions now follow the main-text explanation, so the procedure can be followed without consulting the appendices, which provide derivations, proofs, and implementation details.

We hope this response has addressed your concerns.

AuthorsOpenReview ↗

W2 (1/2): Positioning Relative to Zielnicki et al.

Thank you for drawing our attention to this related work and for encouraging us to clarify the distinction between the valuation objective, credit construction, and intervention mechanism. We follow the same separation below.

Shared objective, different contribution. Zielnicki et al. ask how much demand recommendations and titles generate on a streaming catalog. In Section 5.2 of the current arXiv version they define the engagement lost when a set of titles is removed from the catalog, net of substitution to other titles, and extend the definition to new titles. Our shielding estimand pursues the same objective, stated for next-period utility under the serving policy's reallocation. Our related-work text described their paper too narrowly, and we do not claim the objective as a contribution.

Why a different route is needed. Their value is a counterfactual simulated from a discrete-choice demand system. A user chooses at most one title per day from about 7,000 titles, and each title has its own learned parameters. Their data-scale analysis (Figure 10) shows that this model depends on large estimation samples. In a feed-based short-video ecosystem the inventory is orders of magnitude larger and is renewed daily. A session always consumes items, so supply value shows up in graded outcomes such as watch time, open reasons, and return visits. Exposure is allocated by the ranker instead of being chosen from a menu, so substitution operates through the reallocation of a fixed attention budget. Item-specific demand parameters cannot be learned for most items before the exposure decision is due.

Zielnicki et al.SCSV
Source of item valuesimulation from a structural demand modelcredit settled from production logs, no item-specific demand parameters
Substitutionchoice probabilities, with replacement recommendations imputed from predicted utilitiesexposure reallocated by the production ranker, discount calibrated on randomized shielding
Outcomesame-day probability of watching any titlenext-period ecosystem utility (watch time, open reasons, revenue)
Evidence on removal valuediversion ratios in an experiment that reallocates impressions across categories, no removal experimentrandomized removal of supply across cohorts, doses, and metrics (Section 3)
Public resourceremoval estimates withheld for confidentialitybenchmark pairing production logs with randomized shielding outcomes
Usecatalog optimization, release schedulingupload-time prior, exposure allocation, online deployment (Section 4)
AuthorsOpenReview ↗

W2 (2/2): Positioning Relative to Zielnicki et al.

What the credit construction adds. First, no demand system is fitted. Lagged, orthogonalized attribution and settlement give one signed credit to every item with logged behavior, including the long tail. Second, the replaceability discount is calibrated on the net loss of randomized shielding rounds, in which the production ranker itself performs the reallocation. Third, the unit budget bounds the credits settled within a user-period by that user's realized utility change (Proposition 2.1), an accounting property that a simulated counterfactual does not carry. Fourth, the summed credits of a set form a prediction that a removal experiment can reject. Section 3 tests this prediction on a random 20% of core authors, a value-ranked top 10%, five shielding doses, live revenue, and the promotion direction. It reports every deviation (at most 0.67 percentage points), including the consistent shortfall on watch time. Neither paper validates the removal value of a single item.

What the intervention mechanism adds. SCSV additionally connects settled credits to an upload-time prior, deploys a value-guided support policy, and evaluates its ecosystem effects in a randomized online experiment. For new titles it relies on embeddings built from human-curated tags, a resource that user-generated uploads do not have. SCSV carries the value into allocation. A prior trained on settled credits scores each upload, with its uncertainty, from serving-time features. Allocation is formulated net of the value displaced from substitutes (Section 2.3). The deployed gate and boost is evaluated in a user-randomized online experiment, in which open reasons rise by 0.363% (p = 0.019) and open-reason ROI rises from 13.99% to 50.00% (Table 4).

Summary. Zielnicki et al. define this removal value on a streaming catalog and show that a structural demand model can simulate it. SCSV adds three things. The first is a method that settles, from production logs, an item credit targeting this value. Its replaceability discount is calibrated on earlier randomized removal rounds, and its set-level predictions are tested on disjoint ones. The second is a mechanism that turns the credit into exposure decisions and is evaluated online. The third is a benchmark, released upon acceptance, that pairs production logs with randomized removal outcomes.

Revision. We have corrected the description of Zielnicki et al. in the Introduction and Appendix M.2 and added this comparison as Table 16 in Appendix M.2. The changes are highlighted in blue.

We hope this response has addressed your concerns.

Reviewer_AekyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCSV, a substitution-corrected valuation method for recommender ecosystems: an item's value is defined not by its own engagement but by a shielding counterfactual. To estimate this quantity from production logs, the method combines lagged differencing, double machine learning, a sign gate, a replaceability discount, and unit-level credit budgets, settling attributions into signed credits that are then distilled into uncertainty-aware priors driving cold-start selection and exposure allocation. The authors construct the first industrial-scale benchmark paired with randomized shielding ground truth; set-level estimates agree with shielding losses in sign and ordering, and deployment at 20M+ DAU significantly improves watch time, open reasons, and retention.

Critical strengths
  1. Supply substitution/cannibalization is a core pain point for large-scale platforms.
  2. The experimental design is well-conceived and convincing.
Critical weaknesses
  1. The method involves a complex chain of steps, and the components are quite technical, which has made me somewhat hesitant about contribution to this manuscript.
  2. The estimand covers only next-period response, so metrics with strong within-period responses are systematically under-credited.
  3. Treatment is a share change within a conserved budget. How do the estimand and settlement adapt to settings where total attention is not conserved?
Questions for authors

Please refer to the weakness.

6 comments
AuthorsOpenReview ↗

W1: Clarifying SCSV’s Methodological Design and Contributions

Thank you for your careful reading and thoughtful feedback on the methodological design and contributions of our work. We agree that explaining each component's role before introducing its technical details would help readers understand the overall rationale. We clarify below how the stages fit together and support the research contributions.

SCSV addresses a unified research objective: how to construct a supply-value signal that accounts for substitution under finite attention and use it to guide exposure support. Fig. 1 illustrates the central difficulty: boosting item 2 gains xx but displaces y1+y2y_1+y_2 from items 1 and 3. An item-level engagement measure records the gain xx, while the ecosystem-level net gain is only x−y1−y2x-y_1-y_2. A value signal for supply support must therefore account for the value displaced from substitute supply. As outlined in the contribution paragraph of the Introduction, our contributions can be grouped into three connected aspects.

First, substitution-corrected credit settlement. We take as the valuation target the next-period ecosystem utility lost when an item is removed and its attention is reallocated, and introduce a settlement mechanism that constructs operational item credits targeting this quantity from production logs. Lagged differencing and DML provide the attribution inputs: each item's predicted contribution to a user's utility change. Settlement determines how these inputs become credits. A contribution enters settlement only when its direction agrees with the user's observed utility change (sign gate); its magnitude is discounted by the estimated share that substitutes could recover (replaceability discount); and the sum of absolute item credits within each user-period is bounded by the magnitude of that user's observed utility change (unit budget). The substitution illustrated in Fig. 1 motivates the replaceability discount; the sign gate and unit budget determine which contributions enter settlement and bound the total absolute credit. For each shielded set, credits are computed using the substitutes remaining after that set is removed, and their sum gives a prediction that a randomized shielding experiment can check. This settlement construction is the main methodological contribution, and its aggregate predictive validity is evaluated through the experiments described below.

Second, value-guided supply intervention. Settlement requires observed behavior, whereas new supply must be evaluated before behavioral histories become available. We therefore predict settled credits and their uncertainty using information available at upload time, and formulate intervention decisions that account for finite exposure budgets and the value of displaced supply. The deployed policy is simpler: it uses predicted-credit and uncertainty thresholds to determine eligibility, then passes the value score to the existing exposure-support mechanism. This makes the signal produced by the first contribution usable for selecting and supporting new supply.

Third, evaluation through randomized experiments and deployment. We pair production logs with randomized supply-shielding outcomes and use experiments disjoint from the calibration rounds to compare log-based estimates with set-level removal effects. Large-scale online experiments then evaluate the deployed policy. These two kinds of evidence address aggregate predictive validity and practical decision benefits, respectively.

The stages serve one objective: attribution provides inputs, settlement produces substitution-corrected credits, prediction makes them available for new supply, and randomized experiments and deployment evaluate their effectiveness. The revised Introduction uses the example in Fig. 1 to motivate substitution-corrected valuation and presents three numbered contributions. Section 2 now opens with an overview explaining the purpose and sequence of the four stages before the mathematical details.

We hope this response has addressed your concerns.

AuthorsOpenReview ↗

W2: Next-Period Valuation and Immediate Responses

Thank you for raising this important question about the temporal scope of our valuation. SCSV deliberately focuses on how supply changes affect next-period ecosystem utility under substitution, with the aim of providing a value signal for exposure support (Sec. 2.1). Same-period (within-period) and next-period responses contribute to the full effect over the combined window; our current formulation specifically targets the next-period component.

Purpose and rationale of the next-period target. We seek to connect observed supply changes to subsequent ecosystem outcomes. In our logged-data formulation, the same-period utility change is computed from the same behavior changes used to construct the treatment representation (Sec. 2.2). Regressing that outcome on those inputs would reproduce a mechanical accounting relationship, which by itself does not identify the counterfactual effect of removing an item and reallocating its exposure. Adaptive exposure allocation introduces additional contemporaneous feedback. By relating preceding-period changes to subsequent utility changes under the assumptions in Appendix B, we give the attribution problem an explicit temporal ordering and a defined subsequent outcome to explain.

Existing evidence and practical usefulness. Our two main validation metrics differ in timing. Open reasons are user-initiated entries that follow exposure in the feed, and their settled estimates are within 0.30 percentage points of the randomized shielding effects. Watch time responds within the period. Its estimates have the same sign as the online losses, follow the same ordering across the five shielding ratios, and show the same qualitative convex profile, but are smaller in magnitude at every ratio (Table 1). This pattern is consistent with the next-period scope contributing to the smaller magnitudes for metrics with strong same-period responses.

In a separate online experiment, the deployed policy built on this credit improved view time by 0.053% (p = 0.003; Table 4). This result supports the signal’s decision value in the tested setting: a valuation targeting subsequent ecosystem utility can guide exposure support and improve the measured online outcome.

Revisions and extensions. The revised Introduction now states the next-period target and the reason for it. Section 2.1 has a new paragraph, "Scope of the estimand", and Section 3.1 notes that the offline watch-time estimates target the next-period estimand. Your emphasis on same-period responses also aligns with our ongoing research. We are investigating explicit modeling of these responses, and the Conclusion now mentions this direction as an extension that could broaden the temporal coverage of supply valuation. The changes are highlighted in blue.

We hope this response has addressed your concerns.

AuthorsOpenReview ↗

W3: Attention Conservation and Variable Total Attention

Thank you for asking how the framework extends to settings with variable total attention. We address two kinds of variation in attention in turn: differences in total attention across users and periods, and changes in total attention caused by the intervention itself.

What is conserved in the current setting. Total attention may vary across users and periods: Ca,tC_{a,t} in Sec. 2.2 is the actual attention volume of unit aa in period tt. The conservation condition applies only to the intervention: exposure released by a removed item goes to other items, and exposure added to a boosted item is taken from other items, within the same period's total exposure (App. A, "Allocation mechanism"). It concerns current-period exposure, not next-period utility: changes in next-period watch time, app duration, or return visits are outcomes the estimand is intended to capture.

Estimand. The policy-dependent removal-value formulation in Eq. (1) can be retained for interventions that change total exposure. In that setting, T−iT_{-i} must describe both the reallocation to the remaining supply and the intervention-induced change in total exposure, so the resulting utility difference includes both channels. App. A discusses the boundary case in which the released exposure is not reallocated at all, where the estimand reduces to the conditional expected gross utility difference.

Settlement. The sign-gating rule and the unit-budget normalization can retain their form once the attribution and compensation inputs are adapted to the new policy. The sign gate checks whether an attributed contribution agrees in direction with the unit's realized utility change, and the unit budget bounds the total absolute credit by ∣Ya,t+1∣|Y_{a,t+1}|; neither operation algebraically requires exposure conservation. Two inputs require adaptation. First, the replaceability discount must reflect the utility recovered by the remaining supply under the actual policy response, since the released exposure can no longer be assumed to be fully reallocated to other items. Second, fixed-budget reallocations identify only contrasts between items' marginal effects (App. A). Valuing changes in total exposure additionally requires their common level relative to a specified reference, such as the value of attention that leaves the platform. This calls for suitable variation in total exposure under appropriate identification assumptions, or an explicitly modeled outside option with conditions that identify that reference level.

Revision. We have stated these two kinds of attention variation explicitly in Sec. 2.1, and added a paragraph to App. A ("Interventions that change total exposure") that discusses how the estimand and settlement extend, including the conditions the extension requires. Both changes are highlighted in blue in the revised PDF.

We hope this response has addressed your concerns. If you have any further questions or suggestions, please let us know; we will be happy to respond promptly. We would appreciate your reassessment of our work in light of these clarifications and revisions. Thank you again for your time and consideration.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response: Main contribution and revisions

We thank the reviewers so much for their careful reading and suggestions on the contribution, credit assignment, and evaluation of SCSV. We have revised the manuscript and added component ablations and normalization-sensitivity analyses. Below we summarize the main contribution and changes; detailed responses appear under the corresponding reviews.

Main contribution and practical significance. Under finite attention, an item can attract engagement that other items would have received in its absence. SCSV constructs item-level credits that account for this substitution and depend on the supply remaining after removal. A shared compensation model, calibrated on earlier randomized set removals, discounts log-based attributions according to the utility expected to be recovered by substitutes. Settlement also bounds each attribution unit’s total absolute credit by the magnitude of its realized utility change. During subsequent evaluation, the compensation parameters remain fixed, while the remaining-supply features change across removal scenarios. This enables loss predictions for held-out sets and removal doses without a separate experiment for every item. The credits also serve as training targets for evaluating new uploads before behavioral histories become available, connecting supply valuation to exposure support. In the reported online experiments, the deployed policy increases open-reason intervention ROI from 13.99% to 50.00%, approximately 3.6 times the baseline.

Main revisions.

  • Presentation and positioning. We revised the opening of Section 2 to explain what each stage addresses and how attribution, settlement, upload-time prediction, and exposure support connect before introducing the mathematical details. The Introduction presents three numbered contributions, the settlement illustration now appears in Section 2.2, and Appendix M.2 expands the comparison with structural removal-value models.

  • Component ablations. Following the reviewers’ suggestions, we separately remove the sign gate, replaceability discount, and unit-budget cap, retaining the other two components in each case. We compare these variants against the existing randomized shielding results. Across all four evaluated sets and all five shielding doses, each ablation increases the absolute deviation from the randomized effect estimate, with the largest deterioration occurring when the replaceability discount is removed. These results provide evidence for the role of each component in the tested settings. The setup and results are reported in Appendix J.1 and summarized in Section 3.2.

  • Credit assignment and sensitivity. Appendix B.1 now explains the attribution-weight ambiguity, how the fitted representative is determined, and how this choice can affect credits. We added a sensitivity analysis on production logs across three utility definitions, examining credit signs, rankings, and set-level predictions. We report the full perturbation range and distinguish local agreement in rankings from sensitivity in predicted magnitudes.

  • Scope and interpretation. Section 2.1 and Appendix A clarify the next-period target, the meaning of attention conservation, and the conditions needed to extend the formulation to changes in total exposure. We also explain the distinction between set-level predictive validation and individual-credit accuracy, and between the reference intervention formulation and the deployed policy.

  • Data and evaluation protocol. Appendix H provides additional information on data scale, feature and label timing, daily training and evaluation partitions, experimental cohorts, and observation windows. It distinguishes model-fit monitoring, nuisance cross-fitting, and randomized validation, and specifies that compensation parameters are calibrated on earlier disjoint rounds and frozen before validation.

Blue highlights mark the added analyses and key explanatory revisions. We appreciate the reviewers’ suggestions, which have helped us present the contribution, supporting evidence, and scope more clearly.

AuthorsOpenReview ↗

A kind request regarding the discussion of submission 6075

Dear Area Chair,

Thank you very much for your time and for overseeing the review of our submission 6075, “SCSV: Substitution-Corrected Supply Valuation for Value-Guided Intervention in Recommender Ecosystems.” We are also grateful to the reviewers for their careful reading and constructive suggestions.

We posted detailed responses to each review on 11 November, uploaded a revised manuscript, and provided a general summary of the revisions on 13 November. We followed up with the reviewers on 14 November, but have not yet received further feedback. We appreciate that this is a busy time for everyone involved.

Following the reviewers’ suggestions, we added component ablations against randomized shielding results, an analysis of sensitivity in credit assignment, and further details about the data and evaluation protocol. We also clarified the contribution and the limitations of our item-level claims. The general response and individual replies provide further details and references to the revised manuscript.

If you consider it appropriate, we would be very grateful for your help in inviting the reviewers to take another look at our responses and share any remaining questions or concerns. We would also appreciate your consideration of our responses and revisions in your assessment. We would be very happy to provide any further clarification that might be helpful.

Thank you again for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.