- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how exposure bias affects Shapley data valuation in recommendation systems with implicit feedback. The authors show that naive valuation can systematically favor frequently exposed items and undervalue relevant long-tail items. They propose SHADE, which applies doubly robust estimation to per-point values before computing exact KNN-Shapley values. The paper provides theoretical results on bias propagation, ranking inversion, and exposure feedback dynamics. Experiments on synthetic benchmarks and real recommendation datasets evaluate value recovery, ranking fidelity, and downstream recommendation performance. The results show that SHADE substantially improves long-tail value recovery, although ranking accuracy and downstream benefits vary across settings.
Critical strengths
- The paper identifies an important source of bias in data valuation. The distinction between exposure frequency and underlying relevance is important for recommendation systems that use data values to guide collection or resource allocation. Theorem 5.3 characterizes how exposure bias propagates into Shapley values, while Equation 7 establishes a condition under which exposure differences reverse the ordering of relevant items. These results provide a useful explanation of why valuation on implicit-feedback logs can systematically disadvantage long-tail items.
- The proposed method has clear theoretical properties and preserves efficient computation. SHADE combines doubly robust per-point estimation with the exact KNN-Shapley recursion. Proposition 5.1 establishes the conditional unbiasedness property, and the linearity and contraction results explain how estimation errors propagate through the valuation operator. The method preserves the existing computational structure rather than requiring expensive Shapley approximation. This makes the approach technically coherent and potentially useful for large-scale valuation.
- The experimental evaluation is extensive and includes informative negative results. The paper evaluates synthetic MNAR settings, datasets with randomized-exposure references, and large-scale recommendation data. It also compares against multiple external valuation methods and examines nuisance estimation, candidate-universe selection, and unmeasured confounding. The experiments distinguish value-level recovery from ranking accuracy and downstream performance. The inclusion of negative results, particularly for fixed-log interventions and natural-exposure augmentation, makes the empirical conclusions more credible.
Critical weaknesses
- The methodological novelty is somewhat limited relative to the existing components. The main estimator is based on standard doubly robust estimation, while the KNN-Shapley recursion is inherited from prior work. The proposed method combines these components by correcting per-point values before applying a linear valuation operator. Although this is a useful application, several theoretical properties follow directly from the existing AIPW estimator and the linearity of the Shapley operator. The paper would benefit from a clearer distinction between genuinely new theoretical insights and properties inherited from established statistical techniques.
- The recovered value rankings remain substantially below the oracle in realistic settings. Section 6.1 reports that SHADE improves ranking fidelity over naive valuation, but the remaining gap is considerable. Across several real-data settings, the estimated SHADE Kendall correlation is approximately 0.20–0.26, compared with a true-nuisance ceiling of 0.60–0.90. The nuisance interpolation analysis identifies outcome-model quality as a major source of this gap. This suggests that correcting exposure bias alone is insufficient for reliable point-level attribution. The method improves the overall value distribution, but its ability to identify which individual interactions are most valuable remains limited.
- The theoretical guarantees rely on identification assumptions that may be difficult to satisfy in production recommendation logs. Assumption 1 requires positive exposure probabilities, conditional independence between exposure and relevance, and a candidate universe selected independently of realized exposure. These conditions are important for the doubly robust guarantee but may not hold in recommendation environments with unobserved user preferences or policy-dependent candidate generation. Appendix F.22 examines unmeasured confounding and shows that recovery deteriorates when the assumption is violated. While this sensitivity analysis is useful, the empirical results do not establish how reliably the required conditions can be met in naturally collected production logs.
- The downstream experiments do not establish that the Shapley computation provides a necessary advantage over direct outcome prediction. Section 6.3 reports that directly ranking never-exposed candidates using the outcome model can match or outperform selection based on the full SHADE score. The paper also finds that an outcome-model refit without IPS weighting can produce the same augmentation benefit. This distinction matters because the main downstream gains appear to come from assigning predictions to previously unexposed candidates, rather than from Shapley attribution itself. Although the paper provides evidence that the Shapley layer improves valuation fidelity, its additional contribution to downstream recommendation performance remains unclear.
- Improvements in value recovery do not consistently translate into better recommendation outcomes. The paper reports substantial recovery of long-tail value, but the benefits of acting on these values depend strongly on the intervention. Fixed-log pruning and reweighting do not consistently improve top-line recommendation accuracy, while the natural-log KuaiRand-Pure augmentation experiment produces no significant gain. In addition, the KuaiRec production-log analysis shows that SHADE can over-correct the relationship between popularity and measured value. These results raise questions about when more accurate value attribution leads to better practical decisions. The distinction is acknowledged in the paper, but it limits the generality of the downstream claims.
Questions for authors
- Beyond combining AIPW estimation with the existing KNN-Shapley recursion, which theoretical results should be regarded as the main new contribution? How do these results differ from applying a standard doubly robust pseudo-outcome to another linear valuation operator?
- Given the substantial gap between estimated SHADE rankings and the true-nuisance oracle, how reliable are the resulting valuations for applications that require accurate individual-level rankings, such as data selection or contributor compensation?
- Can the authors clarify how the required exposure and relevance identification assumptions would be assessed in naturally collected recommendation logs? In particular, how should SHADE be used when unobserved factors affect both exposure and relevance?
- In the augmentation experiments, direct outcome-model ranking sometimes outperforms the full SHADE score. What practical decision benefits specifically require Shapley attribution rather than direct relevance prediction?
- Can the authors identify conditions under which improved valuation fidelity should lead to improved downstream recommendation performance? How should the negative KuaiRand-Pure results and the observed over-correction in production logs affect the practical interpretation of SHADE?