acceptodds
Under review as a conference paper at ICLR 2027

Improving Shapley Data Valuation for Recommendation Systems by Debiasing Popularity

Abstract

Data valuation decides which interactions a recommendation system keeps and acts on, and how revenue is split among content providers. A wrong value therefore propagates into both what users see next and who gets paid. In systems with implicit feedback, the logging policy makes such errors systematic: exposure is missing not at random (MNAR), so a point can look valuable because it was shown often rather than because it is relevant. We formalize this intuition by revealing that *naive Shapley valuation* shifts credit from low-exposure to high-exposure items, and can reverse their order when exposure gaps outweigh relevance gaps. To counter this, we propose *SHADE (SHApley with Debiased Estimates)*, which corrects each per-point value with debiased and doubly robust estimates before solving the cooperative game with the closed-form KNN-Shapley recursion. Experiments on synthetic MNAR benchmarks and real recommendation datasets confirm that *naive valuation* consistently under-values the long tail, while *SHADE* recovers much of the oracle level and improves value ranking. The correction also assigns value to items that were never exposed, which enables augmentation and collection strategies that naive valuation cannot support.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Data valuation under MNAR exposure, Shapley values, doubly robust estimation, recommender systems, positive-unlabeled learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PawrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how exposure bias affects Shapley data valuation in recommendation systems with implicit feedback. The authors show that naive valuation can systematically favor frequently exposed items and undervalue relevant long-tail items. They propose SHADE, which applies doubly robust estimation to per-point values before computing exact KNN-Shapley values. The paper provides theoretical results on bias propagation, ranking inversion, and exposure feedback dynamics. Experiments on synthetic benchmarks and real recommendation datasets evaluate value recovery, ranking fidelity, and downstream recommendation performance. The results show that SHADE substantially improves long-tail value recovery, although ranking accuracy and downstream benefits vary across settings.

Critical strengths
  1. The paper identifies an important source of bias in data valuation. The distinction between exposure frequency and underlying relevance is important for recommendation systems that use data values to guide collection or resource allocation. Theorem 5.3 characterizes how exposure bias propagates into Shapley values, while Equation 7 establishes a condition under which exposure differences reverse the ordering of relevant items. These results provide a useful explanation of why valuation on implicit-feedback logs can systematically disadvantage long-tail items.
  2. The proposed method has clear theoretical properties and preserves efficient computation. SHADE combines doubly robust per-point estimation with the exact KNN-Shapley recursion. Proposition 5.1 establishes the conditional unbiasedness property, and the linearity and contraction results explain how estimation errors propagate through the valuation operator. The method preserves the existing computational structure rather than requiring expensive Shapley approximation. This makes the approach technically coherent and potentially useful for large-scale valuation.
  3. The experimental evaluation is extensive and includes informative negative results. The paper evaluates synthetic MNAR settings, datasets with randomized-exposure references, and large-scale recommendation data. It also compares against multiple external valuation methods and examines nuisance estimation, candidate-universe selection, and unmeasured confounding. The experiments distinguish value-level recovery from ranking accuracy and downstream performance. The inclusion of negative results, particularly for fixed-log interventions and natural-exposure augmentation, makes the empirical conclusions more credible.
Critical weaknesses
  1. The methodological novelty is somewhat limited relative to the existing components. The main estimator is based on standard doubly robust estimation, while the KNN-Shapley recursion is inherited from prior work. The proposed method combines these components by correcting per-point values before applying a linear valuation operator. Although this is a useful application, several theoretical properties follow directly from the existing AIPW estimator and the linearity of the Shapley operator. The paper would benefit from a clearer distinction between genuinely new theoretical insights and properties inherited from established statistical techniques.
  2. The recovered value rankings remain substantially below the oracle in realistic settings. Section 6.1 reports that SHADE improves ranking fidelity over naive valuation, but the remaining gap is considerable. Across several real-data settings, the estimated SHADE Kendall correlation is approximately 0.20–0.26, compared with a true-nuisance ceiling of 0.60–0.90. The nuisance interpolation analysis identifies outcome-model quality as a major source of this gap. This suggests that correcting exposure bias alone is insufficient for reliable point-level attribution. The method improves the overall value distribution, but its ability to identify which individual interactions are most valuable remains limited.
  3. The theoretical guarantees rely on identification assumptions that may be difficult to satisfy in production recommendation logs. Assumption 1 requires positive exposure probabilities, conditional independence between exposure and relevance, and a candidate universe selected independently of realized exposure. These conditions are important for the doubly robust guarantee but may not hold in recommendation environments with unobserved user preferences or policy-dependent candidate generation. Appendix F.22 examines unmeasured confounding and shows that recovery deteriorates when the assumption is violated. While this sensitivity analysis is useful, the empirical results do not establish how reliably the required conditions can be met in naturally collected production logs.
  4. The downstream experiments do not establish that the Shapley computation provides a necessary advantage over direct outcome prediction. Section 6.3 reports that directly ranking never-exposed candidates using the outcome model can match or outperform selection based on the full SHADE score. The paper also finds that an outcome-model refit without IPS weighting can produce the same augmentation benefit. This distinction matters because the main downstream gains appear to come from assigning predictions to previously unexposed candidates, rather than from Shapley attribution itself. Although the paper provides evidence that the Shapley layer improves valuation fidelity, its additional contribution to downstream recommendation performance remains unclear.
  5. Improvements in value recovery do not consistently translate into better recommendation outcomes. The paper reports substantial recovery of long-tail value, but the benefits of acting on these values depend strongly on the intervention. Fixed-log pruning and reweighting do not consistently improve top-line recommendation accuracy, while the natural-log KuaiRand-Pure augmentation experiment produces no significant gain. In addition, the KuaiRec production-log analysis shows that SHADE can over-correct the relationship between popularity and measured value. These results raise questions about when more accurate value attribution leads to better practical decisions. The distinction is acknowledged in the paper, but it limits the generality of the downstream claims.
Questions for authors
  1. Beyond combining AIPW estimation with the existing KNN-Shapley recursion, which theoretical results should be regarded as the main new contribution? How do these results differ from applying a standard doubly robust pseudo-outcome to another linear valuation operator?
  2. Given the substantial gap between estimated SHADE rankings and the true-nuisance oracle, how reliable are the resulting valuations for applications that require accurate individual-level rankings, such as data selection or contributor compensation?
  3. Can the authors clarify how the required exposure and relevance identification assumptions would be assessed in naturally collected recommendation logs? In particular, how should SHADE be used when unobserved factors affect both exposure and relevance?
  4. In the augmentation experiments, direct outcome-model ranking sometimes outperforms the full SHADE score. What practical decision benefits specifically require Shapley attribution rather than direct relevance prediction?
  5. Can the authors identify conditions under which improved valuation fidelity should lead to improved downstream recommendation performance? How should the negative KuaiRand-Pure results and the observed over-correction in production logs affect the practical interpretation of SHADE?
Reviewer_mHzJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SHADE, a debiased Shapley data valuation framework for recommendation systems that addresses popularity bias caused by missing-not-at-random (MNAR) exposure. The method integrates doubly robust estimation with the closed-form KNN-Shapley recursion to correct per-point values while preserving computational efficiency. The authors provide theoretical analyses of valuation bias, ranking consistency, and exposure feedback dynamics. Experiments on synthetic and real-world recommendation datasets demonstrate improvements in long-tail valuation and value ranking, although downstream performance gains vary across different exposure settings.

Critical strengths
  1. The paper addresses an important yet relatively underexplored issue in data valuation for recommendation systems: the distortion of Shapley values caused by popularity-biased exposure. The motivation is clear, and the connection between biased data valuation, long-tail underrepresentation, and downstream exposure feedback provides a meaningful perspective on the problem.
  2. The paper presents a coherent theoretical framework for understanding how exposure bias propagates through the KNN-Shapley operator. In particular, the analysis of long-tail valuation bias, rank inversion conditions, and double-robust estimation provides useful theoretical insights. The assumptions are explicitly stated, and the derivations establish a connection between established debiasing estimators and cooperative-game-based data valuation.
  3. The paper considers multiple recommendation datasets, exposure settings, and downstream applications. Beyond valuation accuracy, it investigates exposure feedback dynamics and the effects of corrected valuation on downstream recommendation performance. The authors also acknowledge several unfavorable results, including limited ranking recovery, potential overcorrection, and the absence of consistent downstream improvements under natural exposure. This transparency strengthens the credibility of the empirical analysis.
Critical weaknesses
  1. The core approach primarily combines a conventional doubly robust estimator with the existing closed-form KNN-Shapley recursion. While applying this combination to exposure-biased data valuation is meaningful, the main algorithmic components are established techniques. Moreover, several theoretical guarantees follow from the standard properties of doubly robust estimation and the linearity of the valuation operator.
  2. Although the paper provides additional problem-specific analysis, it remains unclear whether the proposed framework constitutes a sufficiently substantial methodological advance beyond this integration.
  3. Whether the Shapley computation provides meaningful benefits beyond the debiased outcome estimates themselves. As acknowledged in Section 6.3, directly ranking unexposed candidates using the outcome model can outperform the full SHADE score in downstream augmentation experiments. This observation suggests that some practical improvements may primarily arise from the outcome estimation component rather than the Shapley attribution mechanism.
  4. The downstream improvements are largely demonstrated in controlled or semi-synthetic exposure settings, while the natural-exposure experiments on KuaiRand-Pure do not show statistically significant gains. Moreover, the theoretical guarantees depend on assumptions such as conditional independence between exposure and relevance, positivity, and a fixed candidate universe, which may be difficult to satisfy in practical recommendation environments. Although the paper acknowledges these limitations and provides sensitivity analyses in the appendix, the main paper does not sufficiently establish how reliably the proposed approach translates into improvements in realistic deployment scenarios. This limits the strength of its practical claims.
Questions for authors

Q1. Given that the outcome-model-only approach outperforms SHADE in some experiments, what additional benefits does the Shapley component provide? Q2. Under what practical conditions can SHADE achieve meaningful downstream improvements, especially given the limited gains under natural exposure settings?

Reviewer_MWjxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper concerns data valuation on logged recommendation data. The proposed method, SHADE, replaces the per-point value fed into the closed-form KNN-Shapley recursion with a cross-fitted doubly robust (AIPW) estimate, leaving the recursion itself unchanged; this substitution alone provides an MNAR correction at the valuation layer. The paper provides a theoretical analysis and a thorough empirical study. In settings where an oracle valuation is computable, SHADE demonstrates unbiasedness for its estimand, the oracle valuation (label-resampling study, F.2; numerical checks, E.9), and recovers aggregate-level tail value across several datasets. The experiments are impressively thorough and the accompanying discussion is comprehensive.

Critical strengths

S1. The experimental coverage is exhaustive and the ablations are extensive. The paper reports unfavorable findings explicitly (e.g., F.17: the aggregate tail recovery is assembled differently from the oracle's; F.10: ranking UU directly by m^\hat m outperforms the full SHADE score).

S2. Theorem 5.3 decomposes the bias of naive Shapley valuation exactly into an own-term and a cross-term, and gives a closed-form inversion condition for adjacent ranks (Eq. 7). Every closed form and statistical claim is paired with a numerical verification (E.9), so the correspondence between claims and evidence is unusually clear.

Critical weaknesses

W1. The per-point correction does not address the feature geometry, and at realistic scale the positivity it relies on cannot hold in a meaningful form.

(a) The geometry of the feature space that defines the KNN game is insufficiently analyzed. The SVD features are learned from an MNAR interaction matrix (Table 5; F.7, F.11), so popularity plausibly propagates into the geometry itself — the neighborhoods, the sort orders, and hence the oracle s⋆=Aqr\mathbf{s}^\star = A_q \mathbf{r} — through a channel that the per-point value correction does not address. No ablation isolates this channel.

(b) At realistic scale, positivity (Assumption 1(i)) cannot hold in a meaningful form. The feasibility condition of Theorem E.6 (E.10: a feasible budget requires Mπmin⁡≤BM\pi_{\min} \le \mathcal{B}) gives πmin⁡≤B/M\pi_{\min} \le \mathcal{B}/M; at M≈106M \approx 10^6 the admissible floor falls far below the clip level ε=0.05\varepsilon = 0.05 (Table 3), so clipping binds and the correction is capped — a regime the paper itself documents ("true propensities reach 10−310^{-3}, so ε=0.05\varepsilon = 0.05 caps the correction", F.2). Most of the candidate universe then lies effectively off-support, where vDRv^{\mathrm{DR}} degenerates to m^\hat m (acknowledged as "the standard causal-support limitation", E.4). Three consequences: (i) per-pair values become unverifiable extrapolation; (ii) even on the synthetic benchmark, the aggregate tail recovery arises from offsetting errors (F.17: never-exposed points contribute 0.093 against the oracle's 0.183); (iii) on the pair-level selection task over UU (augmentation), the recursion is outperformed by ranking with m^\hat m directly (F.10; Table 10, p=0.003p = 0.003). The claim that never-exposed items receive value therefore reduces, at production scale, to a direct-method extrapolation without guarantees.

W2. The strongest baseline is not reported in the main table, and the primary metrics are not justified.

(a) The strongest feature-plus-label baseline — the DM ablation (propensities enter only as weights in the m^\hat m fit; Algorithm 1, line 5) — reaches τ=0.762\tau = 0.762 against SHADE's 0.5470.547 on the main benchmark (Table 20, γ=0\gamma = 0), yet Table 1 restricts its comparison to "observation-responsive methods" and the DM numbers appear only in Appendix F.2. The strongest τ\tau baseline should be reported in the main table.

(b) The primary metrics are not justified as measures of rank and level fidelity. For rank, τ\tau-vs-oracle favors predictors that do not respond to the observed labels, as the authors themselves note ("DM can rank well without responding to the observed labels", §6). For level, tail share is attainable by a method that uses features alone (Volume: 0.209, Table 18). The evidence that compares values pointwise against the oracle (F.2; the decile-resolved bias of Figure 2; the L1L_1 misallocation of F.18) exists only where an oracle is computable; on natural logs the level evidence is restricted to aggregate shares, where the agreement is imperfect (KuaiRand overshoots to 0.299 against the reference 0.175; the popularity–value coupling is over-corrected to −0.32-0.32).

Questions for authors
  • Q1 (W1a). Can the authors run a feature-geometry contrast — features fit on the fully observed KuaiRec block (exposure-free) versus the production big matrix, with all other components held fixed?

  • Q2 (W2a). Would the authors report the DM ablation in Table 1?

  • Q3 (W2b). Beyond aggregate shares, what level evidence can be offered on natural logs, given the KuaiRand overshoot and the coupling over-correction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.