acceptodds
Under review as a conference paper at ICLR 2027

Listening by the Right Amount: Calibrated Responses to User Requests in Text-Profile Recommenders

Abstract

Natural-language user profiles make recommenders scrutable: users can read what the system believes about them and correct it in words. A correction has a direction and an amount, yet evaluations and training objectives address only the direction, and nothing specifies how much a recommendation list should change. We measure thousands of requests, adding, withdrawing or negating a preference, on users the model never saw, against a principled yardstick for the amount: a user who states a preference should be treated like the reference users who already hold it. Fine-tuned rankers ignore requests (8% of a negated genre’s share removed on MovieLens-1M, 0% on tag-level attributes held out from training). Training them to obey fixes the direction but not the amount: after “I also love ⟨genre⟩ movies”, 98% of the top 20 carry that genre where reference users get 31%, and a withdrawal is executed as a negation; we show that an objective rewarding only the direction never prefers the right amount to a larger one, an error that success metrics cannot see. We propose calibrated edit distillation (CED). During training, a teacher given each request’s kind and attribute shifts the logits of the items it concerns by the log of the population odds ratio, which under a softmax choice model moves a neutral user’s odds exactly to those of the reference users; the model learns to produce this shift from the request’s text alone, and at the optimum only the requested items move. A 125M-parameter CED model removes negated genres completely and 85% of the share of held-out tags it never saw edited, and with a scale selected on validation users it lands within three points of the reference users on added and withdrawn genres and matches its teacher on genres. It follows 36 independently written phrasings once trained on 60 per kind and costs no measurable accuracy, without a separate parser or a generation step.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Controllable Recommendation, Natural-Language User Profiles, Response Calibration, Knowledge Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_t3FQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how text-profile recommendation systems should respond to natural-language user requests, focusing on both the direction and magnitude of preference changes. The authors propose Calibrated Edit Distillation (CED), which uses population-level preference odds ratios to construct calibrated score adjustments and distills these adjustments into a recommendation model without requiring a separate parser. Experiments on MovieLens-1M and MovieLens-20M compare CED against standard ranking models, direction-only training, and other request-following approaches. The results demonstrate improvements in handling preference insertion, withdrawal, and negation, including generalization to unseen attributes and request phrasings, with limited observed degradation in recommendation accuracy.

Critical strengths

The main strength of this paper is its distinction between following the direction of a user request and responding by an appropriate amount. CED provides a principled calibration target based on population preference statistics and integrates it into the recommendation model through distillation. The experiments include multiple baselines, unseen attributes, and unseen request phrasings, providing useful evidence of the method's effectiveness and generalization ability.

Critical weaknesses

The main limitation is that the proposed calibration target relies on population-level preference statistics, which may not fully represent the appropriate response magnitude for individual users. Although the paper includes an evaluation of users with changing preferences, the general validity of this calibration assumption remains insufficiently established. In addition, the performance under unseen request phrasings is less consistent than the main results suggest. For example, in Table 11, CED produces a recommendation share of 0.15 for withdrawn preferences compared with the reference value of 0.22, indicating that the model may still overreact to certain requests. Finally, the claim of no measurable accuracy degradation is based on only three training seeds, with relatively large variation in NDCG@10 changes, making the stability of this conclusion less certain.

Questions for authors
  1. The calibration target is derived from population-level preference statistics. Could the authors further clarify how this target reflects the appropriate response magnitude for individual users? Could the existing evaluation results be used to examine variation in calibration errors across users or attributes?
  2. In Table 11, the response to unseen withdrawal phrasings remains below the reference level (0.15 versus 0.22). Could the authors clarify why calibration becomes less accurate for these requests and whether this behavior is consistent across different attributes?
  3. The reported NDCG@10 changes show relatively large variation across three training seeds. Could the authors further clarify the stability of recommendation accuracy using the existing per-seed results and discuss whether the current evidence is sufficient to support the claim of no measurable accuracy degradation?
Reviewer_qtA8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how text-profile recommenders respond when users make explicit natural-language requests to add, withdraw, or negate a preference. While prior work on controllable recommendation only evaluates whether target items move in the right direction, the authors argue that we also need to calibrate the amount of change. They define the target amount using reference users: a user who adds a preference for an attribute should receive a similar top-K share of that attribute as users whose profiles already state it, while withdrawing a preference should bring the share down to the level of users who do not mention it (and negation should push it to zero). Evaluating on MovieLens-1M (genres) and MovieLens-20M (tags), the authors show that standard fine-tuned cross-encoders largely ignore appended requests. Furthermore, training the model with a contrastive loss (CPC) to enforce direction-only obedience causes severe over-reaction: INSERT saturates the top-20 list (98% of items match the requested genre vs. 31% for reference users), and DELETE wipes out the genre completely (0% share), acting just like a hard negation. To fix this, the authors propose Calibrated Edit Distillation (CED). During training, a privileged teacher uses the true request type and item attributes to shift the factual logits of matching items by the population log-odds ratio, and the student ranker is trained via KL divergence to reproduce this shifted distribution from the text alone. Experiments show that CED matches its teacher on genres, generalizes well to held-out tag clusters and unseen phrasings without needing a separate parser at test time, preserves baseline ranking accuracy, and aligns well with the future preferences of users whose tastes actually changed over time.

Critical strengths
  1. Problem formulation and the distinction between withdrawal and negation: pointing out that direction-only objectives and success metrics reward extreme over-reaction is an important insight for controllable recommendation. In particular, separating preference withdrawal (DELETE, going back to neutral) from aversion (NEGATE, avoiding the attribute) nicely exposes the flaw of standard contrastive training, which collapses withdrawal into complete avoidance. Anchoring the target magnitude to reference users (and validating this against users whose real tastes shifted over time (Table 3)) makes the evaluation benchmark very convincing.
  2. Simple, effective distillation without test-time parsing overhead: framing the structured request and item attributes as privileged information during training and distilling a population log-odds shift into the ranker is clean and well-executed. It avoids the latency and error cascade of parse-then-shift pipelines while generalizing surprisingly well to 164 unseen tag clusters on MovieLens-20M and unseen phrasings.
Critical weaknesses
  1. Cross-encoder scaling bottleneck and reliance on explicit metadata strings: the method is only demonstrated on a cross-encoder that concatenates the user profile, request, and item text and runs a full forward pass for every candidate item. Scoring 500 to 1,000 items per user request is computationally heavy and impractical for full-catalog retrieval compared to two-tower/bi-encoder models or latent bottleneck models (such as TEARS or CCBR) that encode the user profile once. It is unclear if CED can work without token-level cross-attention between the request text and explicit tag lists in the item description.
  2. Gap between theoretical log-odds shift and top-K evaluation (requiring post-hoc scaling), plus single-request scope: Proposition 2 shows that shifting logits by log(kappa_a) (alpha = 1) calibrates the softmax choice probability, but the actual evaluation metric is the top-20 share. Because of this mismatch, the unscaled theoretical shift (alpha = 1) noticeably under-reacts on MovieLens-1M, requiring a scaling hyperparameter (alpha = 2) tuned on validation users (and on MovieLens-20M, only alpha = 1 is reported, leaving a gap on INSERT). In addition, although the paper notes that logit shifts compose additively, all experiments only test single isolated requests on MovieLens.
Questions for authors
  1. Multiple or sequential requests: in Section 3.3 and Proposition 2, you note that logit shifts compose (e.g., INSERT followed by DELETE is the identity). Does the trained CED student model actually preserve this compositionality if a user appends two requests at once (e.g., adding one genre and withdrawing another, or inserting and then deleting the same genre)?
  2. Validation-selected alpha on MovieLens-20M: on MovieLens-1M, tuning alpha = 2 closes the INSERT gap left by alpha = 1. On MovieLens-20M (Table 7, Figure 3), you only report alpha = 1, where INSERT lands at 0.054 vs. the 0.070 reference. Did you run the validation selection for alpha on MovieLens-20M, and does a larger alpha close the INSERT gap on unseen tags without hurting DELETE?
  3. Beyond cross-encoders: can CED be applied to a bi-encoder (two-tower) architecture or text-to-collaborative bottleneck models like TEARS, where the user profile+request is encoded once into a single vector? Or does generalization to unseen tags require cross-attention between the request tokens and the item text?
  4. Factual aversions in the base ranker: Table 5 shows that reference users whose factual profiles explicitly state an aversion still get a 0.20 share of that genre under RANK (almost the same as the 0.22 neutral share). Why does the base ranker ignore stated aversions in factual profiles, and could a similar distillation term help the model respect factual profile aversions too?
Reviewer_cNcTOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies how text-profile recommenders should respond when users add, withdraw, or negate a preference. It argues that following the direction of a request is insufficient: a model may respond by filling or emptying the recommendation list with the named attribute. Calibrated Edit Distillation (CED) constructs attribute-specific score shifts from reference-user behavior and trains the ranker to reproduce the resulting teacher distribution when reading a request. Experiments on MovieLens genres and tags examine response magnitude, negation, locality, and factual recommendation accuracy, with additional tests on unseen attributes, unseen phrasings, other profiles and backbones, and users whose later preferences changed.

Critical strengths
  1. The paper identifies a useful distinction between withdrawing a preference and explicitly excluding an attribute. The examples make clear why high request success alone can hide an undesirable response. Treating response magnitude as an evaluation target is a worthwhile contribution, and the proposed distillation method offers a simple way to incorporate this behavior into the ranker without a separate inference-time parser.

  2. The experimental analysis is unusually thorough for this specific question. The bounded-margin and constant-shift controls, multiple seeds, held-out attributes, independently written phrasings, and alternative profiles/backbones address several plausible explanations for the results. Table 3 also makes a useful effort to connect the proposed target to observed preference changes rather than relying entirely on synthetic requests. These results give me a generally positive view of the work, subject to the interpretation and comparison issues below.

Critical weaknesses
  1. The evidence supports calibration of average attribute exposure more directly than it supports appropriate, useful responses for individual users. Equation (2) takes the absolute difference after averaging recommendation shares, so over- and under-adjustment across users can cancel. The per-attribute results in Appendix C help with aggregation across attributes, but not with variation among users within an attribute. Moreover, the evaluation reference is what RANK recommends to users whose profiles state or omit the preference. This is a reasonable consistency target, but it need not be the amount a particular user wants.

Table 3 is an important check, but its gaps are also differences between means: the emergent-preference result of 0.53 versus 0.55 does not establish that most user-genre pairs are close to their own later behavior. There is a related gap in recommendation utility. Section 2 defines NDCG on the original, unedited profile, so the accuracy results establish preservation of factual ranking, not the quality of the recommendations after a request. Matching a genre's share does not ensure that the selected movies are relevant. Individual-level errors and request-conditioned relevance on the existing Table 3 cohort would help connect the main claim to users' outcomes. I would not require a new large user study to address this concern.

  1. Some conclusions about alternative methods depend on choices in their adaptation. In Appendix G, the CTRL-Rec baseline labels every item as neutral for a withdrawal request and adds the predicted approval to the unchanged factual score. Even a perfect predictor would therefore add the same value to every item and leave the original preference in place. Its difficulty with DELETE is partly built into this labeling scheme. The original CTRL-Rec paper also reports using rank interpolation after finding direct score interpolation less effective (Section 3), whereas this adaptation adds raw scores. Since score scale matters to the argument about a single control weight, this difference deserves a check or a clearer justification.

The parser comparisons are useful and include matched training phrasings in the earlier experiments. However, Table 11's independent-phrasing test reports the 60-phrasing CED model without a learned parser trained on the same expanded set. Including that comparison would make the claim that CED generalizes where parsers fall short more convincing. These issues do not invalidate CED's results, but they limit how broadly I would interpret its advantage over the alternatives.

Questions for authors
  1. On the user-genre pairs in Table 3, does CED reduce the average individual absolute error to each pair's later positive-item share, and what fraction of pairs improve? A distribution or uncertainty estimate would help establish whether the small aggregate gaps are representative, while acknowledging the noise in each user's later ratings.

  2. For that same cohort, how does recommendation relevance change after the corresponding request, measured against later positive items? I am asking about these observed preference changes, rather than evaluating every synthetic request against historical labels that may conflict with the request.

  3. Could you clarify the withdrawal labels and score-fusion choice in the CTRL-Rec adaptation, and whether a comparison with its rank-fusion approach changes the result? A learned parser trained on the same 60 phrasings and tested on the independent set would also help assess the practical alternative.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.