- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an audit framework for evaluating compositional generalization in recommender systems, addressing the problem that group-level metrics cannot distinguish between allocation effects and item-choice quality. Specifically, a model may achieve a higher group-level score simply by allocating more recommendation positions to the target group, without necessarily exhibiting better item ranking within that group.
To address this issue, the authors propose a model-agnostic evaluation method that replaces the recommended items within a group while keeping the number and positions of group recommendations fixed, thereby analyzing the respective contributions of allocation and ordering to overall performance. The paper provides an exact decomposition for Recall with full rankings and derives corresponding bounds for truncated lists. In addition, the authors validate the method through experiments on synthetic data, real-world interaction logs, and audits of existing models. They also demonstrate a group-specific fusion correction strategy based on the audit results.
Overall, the main contribution of this paper is a practical tool for evaluation and diagnosis. The experimental results show that the proposed framework helps identify potential confounding effects in group-level evaluation and provides a basis for analyzing and post-hoc optimizing recommender systems.
Critical strengths
- The paper proposes an evaluation framework with a clear theoretical foundation that is model-agnostic.
The paper identifies an important issue in evaluating compositional generalization in recommender systems: group-level metrics can be affected by both allocation and within-group ranking quality, potentially leading to an overestimation of a model's actual item-selection capability. The proposed audit framework evaluates item-choice quality while keeping the number and positions of group recommendations fixed and provides an exact decomposition for Recall with full rankings. These properties give the method a sound conceptual basis, a clear theoretical foundation, and strong practical applicability.
- The experiments cover a broad range of settings and demonstrate the practical value of the method.
The paper uses synthetic experiments to demonstrate how group-level metrics can be misleading and conducts further analyses across Amazon Electronics, HetRec, Beauty, and Steam. Particularly noteworthy is the application of the audit framework to analyze performance differences between existing models, followed by the development of a group-specific fusion correction strategy that improves overall NDCG while keeping group recommendation positions fixed. These results demonstrate that the framework can support both model-performance analysis and practical improvements to recommender systems.
Critical weaknesses
- The evaluation of compositional generalization on real-world data remains insufficiently rigorous.
In the Amazon Electronics experiments, although the target brand–category combinations are novel for some users, they still appear in the training interactions of other users. Therefore, this setting is closer to user-novel evaluation than to a strict model-novel setting. This limits the extent to which the experimental results support claims about genuine compositional generalization. Although the synthetic experiments validate the proposed evaluation framework, an additional holdout experiment on real-world data, in which the target attribute combinations are absent from the training interactions of all users, would provide stronger support for the paper's claims about compositional generalization.
- The theoretical analysis of NDCG and truncated lists remains insufficient.
The paper provides an exact decomposition for Recall with full rankings. For NDCG and truncated outputs, however, the analysis primarily relies on matched-position comparisons and completion bounds, without establishing a formal decomposition with comparable rigor. Consequently, the theoretical guarantees vary across evaluation metrics and practical recommendation settings. Given the importance of metrics such as NDCG in recommender-system evaluation, clarifying the conditions under which these comparisons are valid and their limitations would strengthen the theoretical completeness of the method.
Questions for authors
- Theoretical guarantees for different evaluation metrics
For metrics beyond Recall, such as NDCG and MAP, could the authors further explain the conditions under which replacing items while keeping group recommendation positions fixed can reliably isolate the item-choice effect? Could the authors provide corresponding theoretical guarantees or explicitly clarify the scope and limitations of the current method?
- Rigorous evaluation of compositional generalization
In the Amazon Electronics experiments, the target brand–category combinations still appear in the training data of other users. Could the authors provide or add a strict model-novel experiment on a real-world dataset, ensuring that the target combinations are absent from the training interactions of all users, to further validate the effectiveness of the proposed audit framework in genuine compositional generalization settings?
- The impact of candidate-pool design
Some conclusions in the paper may be influenced by the design of the candidate pool. Could the authors further investigate the sensitivity of the allocation and ordering decomposition to candidate-pool construction and tie-breaking rules? Is there a standardized candidate-control protocol that could reduce the influence of these factors on experimental conclusions?
- Reliability of conclusions for truncated lists
For short lists generated by generative recommendation models, how tight are the proposed completion bounds in practice? Could the authors provide explicit diagnostic criteria for determining when the conclusions drawn from matched-position comparisons are sufficiently reliable and when they should instead be considered inconclusive?
New strict-holdout experiment, metric guarantees and candidate-pool analysis
W1 / Q2. A strict holdout on real data
Yes, we added the requested experiment in Section 6.3 (Table 1). The Amazon combinations are user-novel, as Section 4.2 of the submitted paper already states. The new HetRec experiment removes every Comedy × UK interaction from training and validation and excludes these films from negative sampling and validation candidates. Comedy and UK remain separately represented in training. We retain the original group-test interactions for 1,926 users. All 266 group films are training-cold and eligible.
The preference-blind rule wins the original group score in expectation, but the feature model chooses better at equal counts on the same users and relevant items. The random value is the exact expectation over shared random group orders. This comparison concerns group recall, not the value of displaced nongroup items.
The feature model also beats personalized attribute similarity (5.69%), a population attribute profile (0.59%) and one shared ranking from the mean learned user vector (5.22%) (Section 6.3.1; Appendix B, Tables 10–12). The shared-ranking comparison is exploratory. This is one combination from a previously examined dataset and does not exclude every shared ranking or identify an internal compositional mechanism. The abstract, introduction and conclusion now include this result, with simpler explanations in response to the writing concerns.
W2 / Q1. NDCG, AP and MAP guarantees
We added Appendix D to derive the NDCG decomposition and state its conditions. Holding group positions fixed keeps the position weights unchanged while items are exchanged. The users, eligible items, relevance, cutoff, ideal-score normalization and tie rules must stay fixed.
For each user, we score each model's group-item order at its own positions and at the other model's positions. These four scores separate changes in positions from changes in choices. Their components sum exactly to the original group-NDCG difference when both orders contain enough items for both sets of positions (Section 3.1; Appendix D). Differences outside the group must also be included to account for full-list NDCG.
With one relevant target, AP is reciprocal rank and the same fixed-weight accounting applies. With several targets, AP credit depends on earlier hits, so that group formula does not apply. Complete hybrid lists can still be scored by AP and decomposed algebraically under a declared construction, including changes in credit for nongroup hits. Appendix D gives the single-target result and a multi-target counterexample. We provide no multi-target AP completion bound.
Q3. Candidate pools and ties
Yes, the components depend on the pool. We added Table 13 in Appendix B, decomposing BPR minus LightGCN Recall@20 on the same 5,004 Logitech users and saved rankings (percentage points):
Within-group orders stay fixed, but pools change counts and the prefix depths compared. Choice is negative in each restricted pool, with pointwise intervals below zero. In the catalogue, allocation dominates, while the choice interval [−0.264, +0.240] leaves its sign unresolved. The confuser-pool gap and random-pool allocation intervals also include zero.
Three alternative shared tie orders leave these recall results unchanged. For the film holdout, even favorable cutoff ties raise personalized similarity only to 5.73%, below every model fit. These are finite sensitivity checks.
Appendix A.7 proposes fixing users, relevance, eligibility, groups, cutoff and a shared tie rule, reporting pool sizes after history masking, and labeling exploratory analyses. This is a proposed protocol, not an established community standard.
Q4. When are incomplete lists conclusive?
For a fixed-cohort mean difference bounded by [L,U], L>0 certifies an advantage for every allowed completion, U<0 certifies the reverse, and bounds containing both signs are inconclusive (Section 3.1; Appendix D).
For TIGER minus SASRec at SASRec's group positions, top-ten bounds are [.004611, .031087] on Beauty and [.000288, .084360] on Steam. Steam's top-fifty output narrows its range to [.001082, .001181] (Section 6.4; Appendix A.5, Table 7). These bounds settle direction even when their width leaves magnitude uncertain. They concern single-target missing ranks. Sampling uncertainty needs a separate interval, and an interval containing zero does not establish equivalence.