acceptodds
Under review as a conference paper at ICLR 2027

Winning Without Composing, Losing While Generalizing: Separating Better Scores from Better Choices in Recommendation

Abstract

Recommendation models may need to suggest items from unseen combinations of familiar attributes, such as brand and product category. We examine whether withholding interactions with selected attribute combinations, while retaining their constituent attributes elsewhere in training, provides evidence of compositional generalization. A score that credits only the withheld group can reward recommending more items from that group rather than choosing better items within it. We introduce a model-agnostic audit that keeps the number and positions of group recommendations fixed for each user while changing which items fill those positions. When both models rank every item, our audit splits the difference in their recall exactly into a part due to how many group items each recommends and a part due to which group items it chooses. In experiments with known user preferences, a preference-blind rule that recommends more items from the withheld combination can win the group score against models that rank its unseen items better. Real-data comparisons reveal both cases where higher performance is driven mainly by recommending more items from the withheld combination and cases where the advantage persists after matching the number and positions of recommendations from that combination. Consequently, our audit distinguishes whether a model recommends more items from the withheld combination or chooses better items within it providing a more informative basis for evaluating compositional generalization in recommendation.

Primary area:
datasets and benchmarks
Keywords:
compositional generalization, recommender systems, offline evaluation, ranking metrics, benchmark auditing, cold-start recommendation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jLnpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an audit framework for evaluating compositional generalization in recommender systems, addressing the problem that group-level metrics cannot distinguish between allocation effects and item-choice quality. Specifically, a model may achieve a higher group-level score simply by allocating more recommendation positions to the target group, without necessarily exhibiting better item ranking within that group.

To address this issue, the authors propose a model-agnostic evaluation method that replaces the recommended items within a group while keeping the number and positions of group recommendations fixed, thereby analyzing the respective contributions of allocation and ordering to overall performance. The paper provides an exact decomposition for Recall with full rankings and derives corresponding bounds for truncated lists. In addition, the authors validate the method through experiments on synthetic data, real-world interaction logs, and audits of existing models. They also demonstrate a group-specific fusion correction strategy based on the audit results.

Overall, the main contribution of this paper is a practical tool for evaluation and diagnosis. The experimental results show that the proposed framework helps identify potential confounding effects in group-level evaluation and provides a basis for analyzing and post-hoc optimizing recommender systems.

Critical strengths
  1. The paper proposes an evaluation framework with a clear theoretical foundation that is model-agnostic.

The paper identifies an important issue in evaluating compositional generalization in recommender systems: group-level metrics can be affected by both allocation and within-group ranking quality, potentially leading to an overestimation of a model's actual item-selection capability. The proposed audit framework evaluates item-choice quality while keeping the number and positions of group recommendations fixed and provides an exact decomposition for Recall with full rankings. These properties give the method a sound conceptual basis, a clear theoretical foundation, and strong practical applicability.

  1. The experiments cover a broad range of settings and demonstrate the practical value of the method.

The paper uses synthetic experiments to demonstrate how group-level metrics can be misleading and conducts further analyses across Amazon Electronics, HetRec, Beauty, and Steam. Particularly noteworthy is the application of the audit framework to analyze performance differences between existing models, followed by the development of a group-specific fusion correction strategy that improves overall NDCG while keeping group recommendation positions fixed. These results demonstrate that the framework can support both model-performance analysis and practical improvements to recommender systems.

Critical weaknesses
  1. The evaluation of compositional generalization on real-world data remains insufficiently rigorous.

In the Amazon Electronics experiments, although the target brand–category combinations are novel for some users, they still appear in the training interactions of other users. Therefore, this setting is closer to user-novel evaluation than to a strict model-novel setting. This limits the extent to which the experimental results support claims about genuine compositional generalization. Although the synthetic experiments validate the proposed evaluation framework, an additional holdout experiment on real-world data, in which the target attribute combinations are absent from the training interactions of all users, would provide stronger support for the paper's claims about compositional generalization.

  1. The theoretical analysis of NDCG and truncated lists remains insufficient.

The paper provides an exact decomposition for Recall with full rankings. For NDCG and truncated outputs, however, the analysis primarily relies on matched-position comparisons and completion bounds, without establishing a formal decomposition with comparable rigor. Consequently, the theoretical guarantees vary across evaluation metrics and practical recommendation settings. Given the importance of metrics such as NDCG in recommender-system evaluation, clarifying the conditions under which these comparisons are valid and their limitations would strengthen the theoretical completeness of the method.

Questions for authors
  1. Theoretical guarantees for different evaluation metrics

For metrics beyond Recall, such as NDCG and MAP, could the authors further explain the conditions under which replacing items while keeping group recommendation positions fixed can reliably isolate the item-choice effect? Could the authors provide corresponding theoretical guarantees or explicitly clarify the scope and limitations of the current method?

  1. Rigorous evaluation of compositional generalization

In the Amazon Electronics experiments, the target brand–category combinations still appear in the training data of other users. Could the authors provide or add a strict model-novel experiment on a real-world dataset, ensuring that the target combinations are absent from the training interactions of all users, to further validate the effectiveness of the proposed audit framework in genuine compositional generalization settings?

  1. The impact of candidate-pool design

Some conclusions in the paper may be influenced by the design of the candidate pool. Could the authors further investigate the sensitivity of the allocation and ordering decomposition to candidate-pool construction and tie-breaking rules? Is there a standardized candidate-control protocol that could reduce the influence of these factors on experimental conclusions?

  1. Reliability of conclusions for truncated lists

For short lists generated by generative recommendation models, how tight are the proposed completion bounds in practice? Could the authors provide explicit diagnostic criteria for determining when the conclusions drawn from matched-position comparisons are sufficiently reliable and when they should instead be considered inconclusive?

1 comment
AuthorsOpenReview ↗

New strict-holdout experiment, metric guarantees and candidate-pool analysis

W1 / Q2. A strict holdout on real data

Yes, we added the requested experiment in Section 6.3 (Table 1). The Amazon combinations are user-novel, as Section 4.2 of the submitted paper already states. The new HetRec experiment removes every Comedy × UK interaction from training and validation and excludes these films from negative sampling and validation candidates. Comedy and UK remain separately represented in training. We retain the original group-test interactions for 1,926 users. All 266 group films are training-cold and eligible.

Group recallFeature model, mean of three fitsRandom rule, exact expectation
Original top ten0.02%3.76%
Ten group films each9.40%3.76%

The preference-blind rule wins the original group score in expectation, but the feature model chooses better at equal counts on the same users and relevant items. The random value is the exact expectation over shared random group orders. This comparison concerns group recall, not the value of displaced nongroup items.

The feature model also beats personalized attribute similarity (5.69%), a population attribute profile (0.59%) and one shared ranking from the mean learned user vector (5.22%) (Section 6.3.1; Appendix B, Tables 10–12). The shared-ranking comparison is exploratory. This is one combination from a previously examined dataset and does not exclude every shared ranking or identify an internal compositional mechanism. The abstract, introduction and conclusion now include this result, with simpler explanations in response to the writing concerns.

W2 / Q1. NDCG, AP and MAP guarantees

We added Appendix D to derive the NDCG decomposition and state its conditions. Holding group positions fixed keeps the position weights unchanged while items are exchanged. The users, eligible items, relevance, cutoff, ideal-score normalization and tie rules must stay fixed.

For each user, we score each model's group-item order at its own positions and at the other model's positions. These four scores separate changes in positions from changes in choices. Their components sum exactly to the original group-NDCG difference when both orders contain enough items for both sets of positions (Section 3.1; Appendix D). Differences outside the group must also be included to account for full-list NDCG.

With one relevant target, AP is reciprocal rank and the same fixed-weight accounting applies. With several targets, AP credit depends on earlier hits, so that group formula does not apply. Complete hybrid lists can still be scored by AP and decomposed algebraically under a declared construction, including changes in credit for nongroup hits. Appendix D gives the single-target result and a multi-target counterexample. We provide no multi-target AP completion bound.

Q3. Candidate pools and ties

Yes, the components depend on the pool. We added Table 13 in Appendix B, decomposing BPR minus LightGCN Recall@20 on the same 5,004 Logitech users and saved rankings (percentage points):

PoolOriginal gapAllocationChoice
Group only−2.5060−2.506
Group + confusers−0.455+1.375−1.829
Group + random−1.827+0.190−2.017
Catalogue+1.363+1.375−0.012

Within-group orders stay fixed, but pools change counts and the prefix depths compared. Choice is negative in each restricted pool, with pointwise intervals below zero. In the catalogue, allocation dominates, while the choice interval [−0.264, +0.240] leaves its sign unresolved. The confuser-pool gap and random-pool allocation intervals also include zero.

Three alternative shared tie orders leave these recall results unchanged. For the film holdout, even favorable cutoff ties raise personalized similarity only to 5.73%, below every model fit. These are finite sensitivity checks.

Appendix A.7 proposes fixing users, relevance, eligibility, groups, cutoff and a shared tie rule, reporting pool sizes after history masking, and labeling exploratory analyses. This is a proposed protocol, not an established community standard.

Q4. When are incomplete lists conclusive?

For a fixed-cohort mean difference bounded by [L,U][L,U], L>0L>0 certifies an advantage for every allowed completion, U<0U<0 certifies the reverse, and bounds containing both signs are inconclusive (Section 3.1; Appendix D).

For TIGER minus SASRec at SASRec's group positions, top-ten bounds are [.004611, .031087] on Beauty and [.000288, .084360] on Steam. Steam's top-fifty output narrows its range to [.001082, .001181] (Section 6.4; Appendix A.5, Table 7). These bounds settle direction even when their width leaves magnitude uncertain. They concern single-target missing ranks. Sampling uncertainty needs a separate interval, and an interval containing zero does not establish equivalence.

Reviewer_hGG8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper considers the problem where a recommendation model learns about different attributes of items separately during training but not a specific combination of these attributes. The authors aim to test if the model can do a good job of recommending items with such a combination of attributes. The authors introduce an audit that measures the performance of models by fixing the number and positions of recommendations from each group. The proposed audit helps determine whether a model just recommends more items from a target group or indeed better items from it. In experiments with synthetic data, a preference-blind rule wins when the all recommendations are considered but learned models win when only items belonging to the target group are considered. The real world data experiments comprise of settings where a better performance is driven mainly by recommending more items from the target group, where this improved performance persists even after matching group positions and where training-cold items are never recommended. Further, the audit also helps introduce a correction. When group-specific combinations of existing rankings are selected using one set of users and the same rules are tested on additional users in one of the datasets, this correction improves mean accuracy over the entire recommendation list.

Critical strengths
  1. The paper provides a model agnostic audit to evaluate compositional recommendations which is helpful to distinguish models that simply recommend more items from a target group, from ones that choose better items from the group.

  2. The experiments include cases where improved performance is a result of higher quantity recommendations and cases where better item choices remain after matching group positions. These outcomes demonstrate that the audit can both challenge or support model advantages.

Critical weaknesses
  1. The writing is difficult to follow. Even though the paper does contain all the relevant details, certain things are not made clear where they are needed. In the experimental write-up, the paper changes multiple settings simultaneously across several sections without a clear way to connect from the previous section. The main message of the paper is easy to lose among all the different model names and experimental settings.

  2. It is difficult to establish a clear relationship between compositional generalization and the audit across the various experiments. The experimentation is substantial but understanding how every experiment relates to the main objective is challenging.

Questions for authors
  1. What is the significance of matched group position evaluation? If a model is really doing a good job, shouldn’t it be able to place the corresponding items at a higher position regardless of the group matching?

  2. What is a high level summary of the goals and results in different experiments discussed and how do they all connect to compositional generalization and the audit?

1 comment
AuthorsOpenReview ↗

Clarifying the audit’s purpose and what the experiments show

Clarification of the summary

The blind rule wins the score that credits only relevant group items in the original catalogue lists. It does not win when all preferred catalogue items receive credit. Section 5 and Figure 2 distinguish these scores. The new film experiment tests group recall, not catalogue recall.

W1. Writing and experimental organization

We clarified the two film setups in Section 4.2, separately introduced the released models in Section 4.3, and divided the synthetic comparisons into subsections in Section 5. The new Section 6.3 explains the strict holdout before its results, and Section 6.3.1 introduces its comparison rules separately.

We made terminology more explicit and consistent across the main text, Related Work, captions and appendices. For example, we replaced ‘native policies’ with ‘original lists’ and ‘group budgets’ with ‘group counts’. Appendix D brings the metric conditions together. These revisions improve readability while preserving the original results.

W2. How does the audit relate to generalization?

The holdout defines what is new to the user or model. The audit asks whether a score advantage reflects better choices within that group or more recommendations from it. The studies demonstrate this distinction, explain existing score differences, and test an application. For that application, we select combinations of existing scores separately for each group on previously examined users, freeze them, and test complete lists on additional Steam users (Section 7). Q2 gives the findings and their different scopes.

Q1. Shouldn't a good model simply place useful items higher?

Yes. However, a group score can also rise without better choices, simply because more recommendations come from that group. Matching opportunities tests which explanation accounts for the advantage. It supplements the original score.

We added the strict film holdout in Section 6.3 (Table 1), removing all Comedy × UK interactions from training and validation. On the same 1,926 users and relevant items:

Group recallFeature model, mean of three fitsRandom rule, exact expectation
Original top ten0.02%3.76%
Ten group films each9.40%3.76%

The random rule wins the original score without using preferences. With ten group recommendations each, the feature model chooses better. Its original lists average only 0.0156 group films, hiding those useful choices. The random value is an exact expectation over shared random orders.

More group recommendations can themselves be useful, so we report allocation alongside choice. Equal counts suffice for recall. NDCG also rewards earlier ranks, so its comparison preserves absolute group positions while exchanging the items filling them (Section 3.1). The group score does not measure the value of displaced items outside the group.

Q2. What do the experiments show together?

Synthetic data show a preference-blind rule winning group recall while feature-aware models choose better at equal counts and win catalogue recall (Section 5). The new strict Comedy × UK holdout shows the group-score versus choice contrast on real data with the combination absent from every user's training interactions (Section 6.3). The feature model also beats the three tested attribute-based or shared-ranking alternatives (Section 6.3.1). The shared-ranking comparison is exploratory.

The existing-model comparisons give different answers. Extra group recommendations explain most of one Amazon advantage (Section 6.1). The partly cold film study's original lists contain no cold films, revealing nothing about their ordering (Section 6.2). TIGER's published advantage on history-defined groups survives matched positions on Beauty and Steam (Section 6.4). The audit can support a winner as well as challenge one.

The application improves complete-list NDCG@10 on additional Steam users by .001318 over a selected global combination, about 0.85%, and remains beneficial under four tested arrangements of group positions (Section 7; Appendix A.6, Table 9). These results were already submitted. Repeat-target users lose on average, and Beauty's selection-stage comparison showed no established additional benefit.

The audit measures ranking behavior, not an internal compositional mechanism. The abstract, introduction and conclusion now include the new film result, with simpler descriptions of the audit and correction in response to the writing concerns.

Reviewer_HBgtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the interpretability of group-level metrics in the evaluation of compositional generalization in recommender systems. The authors point out that a model's recommendation score on a particular item combination is affected both by how many recommendations are allocated to that group and by which items are selected within the group, so a higher score does not necessarily imply better compositional generalization. To this end, the paper proposes a model-agnostic evaluation method that controls the number and positions of recommendations for the target item group in order to separate the contributions of these two factors, and validates it on synthetic data and several real recommendation datasets. In addition, the authors use this evaluation method to construct a rank-fusion strategy that differs across item groups.

Critical strengths
  1. The research question is thought-provoking. The paper identifies an easily overlooked issue in compositional-generalization evaluation: a gain in recommendation score may come from the target item group receiving more exposure opportunities rather than from the model genuinely having a better ability to choose items. This analysis helps re-examine existing conclusions about the generalization of recommendation models.

  2. The evaluation method is simple and reasonably general. By crossing the within-group orderings and recommendation counts of different models, the authors decompose the Recall difference into an allocation part and an item-ordering part. The method does not rely on a particular recommendation architecture, and the authors further discuss feasible bounds for generative models for which complete rankings are unavailable, which adds to its generality.

  3. The experiments support the central claim from several angles. The synthetic experiments use known user preferences to demonstrate the spurious advantage that can arise from blindly increasing the number of recommendations from the target group; the real-data experiments show how models differ in allocation and in item selection. In addition, the authors validate the group-wise ranking correction on a separate set of users that did not participate in rule selection, which strengthens the practical relevance of the work.

Critical weaknesses
  1. The evaluation results do not yet directly demonstrate that a model possesses compositional generalization ability. The proposed method can determine whether a model selects more relevant items under the same recommendation opportunities, but better within-group ordering may still stem from item popularity, existing interaction patterns, or other statistical features rather than from an understanding of attribute combinations. I suggest either restricting the claims to the level of item selection, or introducing further controlled tests that distinguish genuine compositional generalization from other possible prediction shortcuts.

  2. The experimental scenarios are not sufficiently unified. The synthetic data mainly study combinations that never appear in training; the Amazon and HetRec experiments involve, respectively, combinations that are unseen for a subset of users and a near-cold construction; and Beauty and Steam reuse the generalization categories defined by an existing task based on the historical interaction graph. These scenarios differ noticeably, and the paper does not draw a unified conclusion from them. I suggest stating more explicitly what claim each experiment supports and how the experiments relate to one another.

  3. The proposed ranking correction is still insufficiently validated. The group-wise correction improves over the global fusion strategy on the Steam dataset, but the relative gain is small, and the same advantage could not be established on the Beauty dataset. In addition, the current experiments fix the positions that the original model assigns to each item group (a fixation that is necessary for separating the two factors), so they do not yet show that the method remains effective in a complete system that is allowed to reallocate recommendation positions. I suggest validating it on more datasets and in settings where positions may be reallocated.

  4. Some experimental conclusions and statements could be made clearer. The paper uses concepts such as allocation, ordering, item choice and composition, and some definitions and their conditions of applicability are spread across different sections and the appendix. For example, model A and the allocation component A share the same letter, which may cause some ambiguity for readers. Moreover, practical generative recommendation models often provide only candidate lists of limited length, so complete rankings are difficult to obtain; although the paper already provides completion bounds, it does not state when these bounds suffice for a decision, so the operational applicability of the evaluation method in real systems still needs clearer discussion.

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Response on claim scope, experimental evidence and the correction

W1. What does better item choice establish?

The audit tests whether a score advantage reflects better item choices at matched group positions. It does not identify the model's internal computation. Section 2, Definition 3, clarifies this distinction. We added controls for simpler explanations.

The new real-data experiment in Section 6.3 removes every Comedy × UK interaction from training and validation and excludes these films from negative sampling and validation candidates. At ten group recommendations, the feature model averages 9.40% recall across three fits. Similarity to each user's mean training-film attributes gives 5.69%, a population attribute profile 0.59%, and one shared ranking from the mean learned training-user vector 5.22% (Section 6.3.1). The random-order expectation is 3.76%.

Training interaction counts are zero for all group films, but shared attributes can still carry popularity signals. The learned shared ranking recovers about 55% of the feature model's recall. User-specific rankings add 4.19 percentage points, with a conditional 95% interval [3.85, 4.54] and a positive difference in each fit. Favorable cutoff ties raise the personalized similarity rule only to 5.73%, below every fit.

The shared-ranking comparison is exploratory and its interval is unadjusted, conditional on three fixed fits. A common list chosen using test answers reaches 32.98%, so we do not exclude every shared ranking. This is evidence against the tested explanations on one withheld combination (Appendix B, Tables 10–12).

Appendix A.5 also now reports an existing training-count control: TIGER beats ordering by training-event counts at its group positions on both Beauty and Steam. This uses previously examined users, not a new confirmation sample. The abstract, introduction and conclusion now include the new film result, with simpler explanations of the audit and correction in response to the writing concerns.

W2. How do the studies fit together?

They ask whether higher group scores reflect better choices or more group recommendations, while keeping the definitions of novelty separate.

  1. Synthetic data (Section 5) and the new strict film holdout (Section 6.3) show a preference-blind rule winning group recall while feature-aware models choose better at equal group counts.
  2. The existing-model studies explain or test score advantages. Extra group recommendations explain most of one Amazon advantage (Section 6.1). The partly cold film study's original lists never recommend cold films (Section 6.2). A published advantage survives matched positions on Beauty and Steam, whose groups depend on user histories (Section 6.4).
  3. The Steam application tests whether group-specific ranking combinations, selected on previously examined users and then frozen, improve complete recommendation lists on additional users (Section 7).

We clarified the two film setups in Section 4.2, separately introduced the released-model study in Section 4.3, and separated the synthetic comparisons in Section 5. Section 6.3.1 introduces the new comparison rules separately.

W3. Small gain, datasets and changes in positions

The submitted paper already tests the frozen correction under four arrangements: TIGER positions, SASRec positions, group blocks and extra group positions. Counts also change with SASRec positions and extra slots. Gains over the selected global combination are +.001318, +.001236, +.000926 and +.001499 NDCG@10, each with a positive adjusted lower bound (Appendix A.6, Table 9). Positions match between choice rules within each test but change across tests, without retuning. Section 7 now makes these tests explicit.

These are the same additional Steam users, not four independent samples or a system that jointly optimizes allocation and choice. The primary relative gain is 0.85%. Repeat-target users lose on average, and extra-position lists have lower absolute accuracy in a descriptive comparison. Additional-user confirmation was limited to Steam. On Beauty, selection-stage testing did not establish an additional benefit. We present a demonstrated application of the audit, not a generally superior recommender.

W4. Notation and decisions from incomplete lists

We use Δalloc\Delta_{\mathrm{alloc}} and Δchoice\Delta_{\mathrm{choice}} to distinguish components from model AA, clarify notation in Sections 2–3, and collect metric conditions in Appendix D.

For a difference bounded by [L,U][L,U], L>0L>0 certifies an advantage for every allowed completion, U<0U<0 certifies the reverse, and bounds containing both signs are inconclusive (Section 3.1; Appendix D). For TIGER minus SASRec at SASRec's group positions, top-ten lower bounds are positive on Beauty (.004611) and Steam (.000288). Steam's top-fifty output narrows its range to [.001082, .001181] (Section 6.4; Appendix A.5, Table 7). These single-target bounds address missing rankings, not sampling uncertainty.

Thank you for the rebuttal. The authors have addressed my concerns, and I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript and reproduction code

Our revised manuscript was uploaded on 14 November and is available under “Revisions” on the submission page. New and revised text is marked in blue.

The anonymous repository now also includes code and reproduction instructions for the Comedy × UK holdout experiment and the additional Logitech candidate-pool and tie analyses described in our responses and revised manuscript. The README links directly to these instructions. This code update does not change the reported results or conclusions.

https://anonymous.4open.science/r/recommendation-audit-review-26F1/README.md

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.