- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies collective action in recommender systems, but with real existing users instead of identical fake accounts. Authors describe a collective with two properties, cohesion (how similar members are before acting) and coordination (how much they move together toward a shared strategy). With an ALS recommender, they show how such a collective changes the target item, and that the usual clone setting is a special case. They also show that when members keep their diversity, a larger collective is not always better. Experiments on synthetic data and MovieLens are used to claim that collectives do not need to act like clones, and more coordination is not always helpful.
Critical strengths
- The decomposition in Theorem 1 is simple but useful. It shows that, for one ALS item update, the collective enters only through its post-action mean and residual covariance, so the classical clone result and the heterogeneous case are put in one formula. The counterexample in Appendix D.2 is also built carefully, with A and b actually realizable, which makes the point that collective size alone does not decide success concrete rather than only intuitive.
Critical weaknesses
-
The paper does not show that the experiments really reflect the parameters of the model. Users can only change their ratings, not their representations, but the paper does not explain how cohesion and coordination are turned into rating changes, and the actually realized values are not reported. So, when the results say more coordination hurts or heterogeneity is not a penalty, we cannot know whether this comes from coordination or cohesion itself, or just from how the ratings were modified. In addition, one main claim is true by construction. The authors claim that relaxing the clone assumption closes the gap between the optimized strategy and simple heuristics, but this result is shown at zero coordination, where in the paper's own theory the chosen strategy has no effect at all. So, all strategies must look the same there, and this does not tell anything about the real advantage of optimization.
-
The measured influence is mostly the collective influencing itself. The downstream metrics are averaged over all users including members, and in the synthetic setting the Top-10 change reaches almost 0.9 when the collective is around 90% of users. This means most of the effect is members getting recommended the item they just rated. But the point of collective action is to change what other users see. Without results on non-members, and a simple control that adds the same amount of ratings without the target, it is hard to say how much real influence the collective has.
Questions for authors
-
Users can only change their ratings, so how are the requested cohesion and coordination levels actually turned into rating changes? Could you report the realized levels measured after this step, for the main settings in Figure 3? Also, when every member simply adds the same ratings to their own history, moving more toward the strategy and keeping less of the original diversity seem to happen together. How are settings with full coordination and full diversity at the same time realized?
-
Could you report the downstream results only on non-members, and compare with a simple control where members add the same amount of ratings without the target item? This would show how much of the effect is real influence on other users, and how much comes from the members themselves or from retraining noise.
-
In the synthetic setup, is the cohesion level set before the ratings are generated, or only on the learned representations? Related to this, at the smallest collective size in Figure 3(b) the collective has only one user, so a non-zero diversity level does not seem possible. How is this point produced?
-
The theory assumes members did not rate the target item before acting. How many users rated the target item in the synthetic data, and how is this assumption kept when the collective becomes very large? Also, which target item is used for MovieLens?
-
In Theorem 1, the effect of the collective depends on how far the shared strategy currently under-predicts the target item. Since each heuristic strategy is built from a profile that already gives the target a high rating, this gap may be small by design.
3 comments
Thank you for the response. Some answers did not fit the paper, so I read the anonymous code linked in the Reproducibility Statement. It disagrees with several rebuttal statements and shows flaws in the experimental design. I list only points that can be checked directly.
Rebuttal vs. released code
- Rating protocol. The rebuttal says existing ratings stay unchanged and members have not rated the target. In the code, strategic ratings overwrite a member's existing ratings on those items, and a member's earlier target rating is replaced by 5.
- Realized , . The rebuttal says the experiments show realized values close to the requested ones. The paper reports none. The code already records them in every run, yet your reply to sAWT says these experiments are still to be run.
Experimental design
- Item-Shift Success. The code stores only the Theorem 1 formula evaluated at the requested and . The shift measured after refitting is computed but not stored, although Sec. 5.1 describes it as evaluated after refitting. If the Item-Shift panels plot the stored value, they cannot test Theorem 1.
- Definition of . is fit to ratings that include the target at 5, so the gap is small by design, as you confirmed in Q5. Theorem 1 then gives near-zero success at , , so the decline with in Fig. 3(d) follows from the definition of .
- Top-10 Change. It ranks items a user has already rated and averages over all users, so members who just rated the target 5 are included in the exposure count.
- Fig. 3(a) vs. 3(d). Panel (a) at and panel (d) at are the same setting, yet Item-Shift Success is roughly 0.4–0.5 in (a) and below 0.1 in (d).
I have changed the visibility of all my comments in this thread to Everyone. Several of the authors' replies are currently visible only to the committee and reviewers. For a fair and transparent evaluation by a wider audience, I encourage the authors to set their replies to Everyone as well, so that the full discussion can be read in context.
Response to Reviewer JLZg
Thank you for your detailed review and helpful comments. We respond below. (W = weakness, Q = question.)
W1: Please see response to Reviewer sAWT. We show in our experiments that our “realized” gamma and beta are relatively close to the hyperparameters theorized by the model. Re: trivial claim, thank you for your observation; we will remove that claim.
W2: We will run experiments and provide an update.
Q1: We distinguish between tr(ΣC) , the initial heterogeneity, and γ the retention of this heterogeneity during strategic action. Recall from Eq. 2 that the post-strategic representation of a user i can be represented as ui′=(1−β)ui+βuˉC+γϵi. In the case where the collective has complete coordination, this would mean that all the collective members would move towards uh but the persistence in diversity would indicate that their individual deviations from this centroid would be scaled by some factor γ. Intuitively, we introduce this notion our framework to consider the situation in which users are potentially not able to access all of the exact same strategic items but a more formal characterization of this will be left for future work.
Q2: We will run experiments and provide an update.
Q3: The cohesion level is fully set before the ratings are generated. The full flow is as follows:
Q4: In the synthetic data we simulate an environment where the collective members have not rated the target item. However, for each seed we randomly generate the number of non-collective users who have rated each item. For MovieLens, we selected an item with “medium-low” popularity for each seed. We calculated this by applying a log_5 binning procedure to the rate interaction counts associated with each item following [1]. We will happily document and ablate on this protocol for the final camera ready version of our paper.
[1] Salganik, R., Diaz, F., Farnadi, G. (2024). Fairness Through Domain Awareness: Mitigating Popularity Bias for Music Discovery. In: Goharian, N., et al. Advances in Information Retrieval. ECIR 2024. Lecture Notes in Computer Science, vol 14611. Springer, Cham.
Q5: Yes, that is correct. In the future we will augment our experiments with a more diverse set of synthetic settings.