- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates information manipulation in recommender systems, focusing on scenarios where edited item text or manipulated content is used to push the same item to large numbers of users. It measures how often an item is shown only because of manipulation and aims to keep this below a set threshold with high probability, without knowing which item is targeted. CREST removes suspicious records, calibrates the remaining promotion on clean/attacked examples, and then picks the final top-K lists under that limit.
Critical strengths
S1. The problem framing is useful. It targets repeated promotion across users, and works with frozen LLMs without knowing the target. Treating whole records as the unit to remove is a clean idea others could reuse.
S2. The theory appears correct and its assumptions are clearly stated. So my concerns are about what the guarantee means and how it is tested, not about errors in the proofs.
Critical weaknesses
W1. I agree with the motivation, but I'm not sure such a complex method is necessary. The paper uses calibration to prevent over-promotion of specific items. However, under certain conditions, the final selection reduces to a simple rule: the same item can be recommended at most twice across 20 recommendations. If so, it is unclear why complex statistical calibration is needed.
W2. The method also seems too expensive for real-world deployment. To identify influential information, it removes records one at a time and repeatedly queries the LLM. However, the reported inference time excludes the cost of these additional LLM calls. Thus, while the final selection is fast, the efficiency of the overall system remains unclear.
Questions for authors
Q1. Why is such a complex and expensive approach necessary? Wouldn't a simple rule limiting how often each item is recommended be sufficient? Please compare against the base model with the same frequency limit across all settings, and report how often the limit is triggered.
Q2. Do you think the proposed method is practical for deployment in a real commercial system? If so, what evidence supports this?
4 comments
W2/Q2. “… too expensive for real-world deployment … what evidence supports this?”
The complete runtime comparison in our response to Reviewer 5sEX (W4) includes record filtering and its additional LLM calls. As observed in that comparison, CREST’s total runtime, including filtering, is 20.856 seconds on Beauty and 18.227 seconds on Steam, adding 0.748–3.143 seconds and 1.143–1.911 seconds, respectively, relative to the evaluated defenses under the evaluated caching configuration. It achieves this by limiting redundant inference by reusing rankings for unchanged prompts and passing the selected diagnostic rankings directly to final selection (Algorithm 1; Appendix H.4).
Additionally, to assess CREST’s practicality under varying recommendation workloads and manipulation conditions, the extended Amazon Beauty experiments reported in our response to Reviewer 5sEX (W5) evaluate longer sequences, multiple attack targets, and different candidate distributions. The results show that the mean promotion remains below 0.10 across all three variations, supporting CREST’s applicability in more practical recommendation settings.
As evidenced above, CREST offers a practical way to establish promotion control in *commercial batch recommendation workflows while retaining the existing frozen LLM. For platforms where shared item information can influence recommendations across many users, operators can specify a promotion budget, and CREST applies screening and calibrated selection without requiring attack-target identities or manipulation labels. Screened recommendations remain unrestricted whenever calibration permits, allowing relevant items to continue reaching multiple users while controlling repeated manipulation and supporting platform trustworthiness.
Thank you again for the valuable questions and suggestions, and we have updated the repository accordingly. We sincerely believe we address the concerns.
Thank you for the additional experiments. While they are helpful, my main concerns remain.
W1. The new comparison conflates the effects of screening and calibration. Because the cap is active in most blocks, CREST largely behaves like screening combined with a fixed cap, meaning the promotion bound stems primarily from the cap itself. You also note that Without Screening performs similarly to the fixed cap, which suggests that the gains come from screening rather than calibration. Furthermore, since the calibrated threshold is shared across sequences, it caps all items equally when active without distinguishing legitimate repeats from manipulated ones. Comparing CREST against screening with an always-on cap would clarify whether calibration is actually necessary.
W2. Because screening re-queries the LLM for every removed record, the reported computational overhead seems surprisingly low. Could you clarify whether this overhead was measured without caching, and report the average number of LLM calls required per sequence?
For now, I will maintain my current rating.
Thanks for your valuable and prompt feedback. Below, we address the remaining concerns:
W1. “… effects of screening and calibration … comparing CREST against screening with an always-on cap …”
Thank you for this interesting observation. We agree that, while on the surface the analysis seems simply to show that the cap is active in most blocks, the experiment highlights an important characteristic of our framework. As you note, screening contributes to the recommendation-quality gains. This reinforces its importance within CREST, given our setting where manipulated records and target items are unknown.
Additionally, an important distinction we would like to highlight is where and when the cap is applied. Our reported comparison already uses an always-on fixed-cap baseline, applied directly to the unscreened backbone rankings. CREST, on the other hand, applies final selection to the rankings produced by our screening. Applying a cap after this screening therefore already incorporates an important contribution of the proposed framework.
We would also like to highlight that calibration still contributes in these settings. In the reported experiment, 5–40% of blocks remain uncapped, allowing screened recommendations to be retained without frequency restrictions under the promotion guarantee. This percentage varies across the evaluated dataset–attack settings and reflects the promotion remaining after screening. An always-on cap would impose the frequency constraint in every block.
To empirically assess this contribution further, we compare Screening + Always-On Cap against Full CREST, holding screening fixed. We use the Amazon Beauty instruction-injection setting:
| Method | Recall@5 (%) | NDCG@5 (%) |
|---|---|---|
| Screening + Always-On Cap | 12.95 | 7.12 |
| Full CREST | 13.45 | 7.41 |
With the same screening, Full CREST improves both Recall@5 and NDCG@5. These results support calibration’s additional recommendation-quality benefit beyond screening: permitting preserves useful recommendations that an always-on cap would otherwise remove.
W2. “… measurement of overhead… average number of LLM calls required per sequence?”
We agree that screening introduces additional LLM inference as a trade-off for assessing the influence of records on recommendations. CREST avoids repeated computations by storing and reusing previously computed rankings. The reported runtimes do use caching. In our evaluation on Beauty, with diagnostic rankings already cached, screening required an average of 0–1 additional fresh LLM generations per 20-decision sequence.
This caching comes naturally to CREST as it uses it at three specific steps in its screening and final selection procedures. First, testing a record’s omission changes only prompts containing that record; unaffected decisions reuse their original rankings. Second, at the one-omission depth used in our experiments, the screening search reuses the candidate-omission rankings obtained during influence assessment. Finally, the selected omission’s diagnostic rankings already contain the recommendations after removal, so final selection uses them directly without another generation pass (Algorithm 1).
In practical batch recommendation, diagnostic rankings can be generated during batch preparation and reused throughout screening, search, and final selection. The initial inference cost is thus paid only once while the later stages can reuse the resulting rankings.
We sincerely hope to address the concerns. Thank you again for the valuable questions.
Thank you for the valuable assessment of our work and constructive questions. We appreciate the recognition of the useful problem framing, the methodology, and the theory's correctness. Below, we address the concerns:
The motivation of our work is to limit manipulation-induced promotion while preserving legitimate recommendations. During inference, the challenge is that we know neither which items are targeted nor which users’ recommendations have been affected, and clean counterparts are unavailable.
While a simple frequency cap controls promotion by restricting all appearances of an item, it cannot distinguish legitimate repeated recommendations from manipulation-induced promotion. For example, an item genuinely suitable for ten users would still be limited to two appearances, even if none of those recommendations were the result of manipulation.
To address this, CREST evaluates records through their influence on recommendation rankings and uses calibration to account for the remaining uncertainty about promotion before selecting the final recommendations. Calibration provides a statistically justified condition for avoiding unnecessary frequency restrictions. Specifically, as established in Appendix D.2, CREST retains the screened recommendations without a frequency restriction when the calibrated allowance is within the permitted promotion budget. When that allowance exceeds the budget, constrained selection enforces a frequency cap derived from the prescribed promotion threshold and sequence length, ⌊Tη⌋ (equal to 2 for T=20 and η=0.1, as noted by the reviewer), while minimizing screened-rank cost.
To evaluate this empirically, we compare CREST against the backbone with the same frequency limit across all three datasets and manipulation settings. We report the percentage of calibration blocks in which CREST activates the cap and, for both methods, the proportion of test sequences whose recommendations are changed by the cap.
Recall@5 and NDCG@5 entries report fixed cap / CREST.
As observed in the table, CREST achieves higher Recall@5 and NDCG@5 across all nine dataset–attack settings than applying the same frequency cap directly to the backbone. An interesting observation we noticed is that applying the frequency cap indiscriminately gives nearly the same recommendation quality as Without Screening on Amazon Beauty under instruction injection (Table 2), suggesting indiscriminately applying a fixed cap offers little additional utility over selection without screening.
Additionally, calibration leaves the cap inactive in 5–40% of calibration blocks*, allowing screened recommendations to be retained without a frequency restriction when the calibrated bound permits it. Across these settings, the cap modifies 7–27 percentage points fewer test sequences than the fixed-cap baseline. Together, these results support the combined benefit of screening and calibrated selection in preserving useful recommendations.