acceptodds
Under review as a conference paper at ICLR 2027

PREMO: Complementary Training For Personalized Preference Discovery

Abstract

Personalized decisions require discovering query-relevant user preferences from long, noisy histories. These preferences can be summarized in a memo that covers relevant preferences while remaining faithful to historical evidence. However, the complete set of relevant preferences is unobserved, making memo discovery difficult to supervise. We introduce PREMO (PREference MemO), which alternates a single model between two complementary roles. As a proposer, it discovers new atomic preferences and is rewarded by independent verification against the history. As a memo policy, it generates a memo from the query and history alone, rewarded for covering verified preferences without asserting rejected ones. Proposer training expands the verified supervision and directly trains preference recognition, while memo training learns to express these preferences in a useful summary. Our theoretical analysis identifies conditions under which proposer training followed by memo training improves both coverage and faithfulness over memo-only training. We evaluate PREMO on shopping, movie recommendation, and everyday decisions. A trained 4B model writes memos of about 170 tokens that effectively summarize histories 40 to 100 times longer. It improves coverage and faithfulness over memo-only training and raises downstream accuracy by up to 3.7 points. Its memos also match or beat its frontier-model teacher’s own: PREMO covers more and raises downstream accuracy by up to 6.8 points, even when GPT-5.4 itself decides. The advantage over memo-only training holds at 35B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-powered agent, Personalization Agent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6zSiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The paper studies query-conditioned preference discovery from long & short noisy user trajectories and proposes PreMO\texttt{PreMO} to produce a compact preference memo that a downstream task-facing model can use for personalization. The framework can be viewed as a structured teacher-guided preference-learning pipeline where the student is first asked to imitate frontier model-produced demonstrations, then learns to propose additional preference cues under the same frontier model acting as verifier to guide the student model, and finally the student model is expected to generate task-facing memo independently from the history and the query. The methodology separates preference recognition from preference expression, as the proposal stage is intended to improve recognition of supported and query-relevant preferences, and the memo stage learns to express those preferences in a concise and usable form. The paper evaluates both intermediate preference reasoning through coverage, faithfulness, grounding, and proposer acceptance, and downstream personalization through product/movie selection and personalized response generation.

Critical strengths

1. Structured Decomposition of the Approach

The explicit decomposition of preference learning into teacher-guided initialization, verifier-guided preference discovery, and finally independent memo generation is an interesting way of addressing personalization over long histories. Meaningful separation of between preference recognition and expression is useful and evaluation focusing on understanding the model's ability to identify history-supported and query-relevant preference information before expressing it in memo is thoughtful and appreciable.

2. Interesting Results Pattern

The empirical analyses reveal meaningful behavior. The proposer and memo phases affect recognition, coverage, faithfulness, and grounding differently, and the experiments provide evidence that proposer training teaches something that memo-only training does not capture. The matched proposer-vs-memo training comparisons, multi-round preference discovery analysis, and verification-path ablations make internal behavior interpretable.

Critical weaknesses

1. Lack of Evaluation Coverage and Claim Scope wrt Personalization\textit{Personalization} (Primary Blocker)

  • Downstream evaluation coverage is incomplete relative to broader personalization positioning. The main comparisons are against alternative history, retrieval, memory, and memo-trained approaches consumed by generic LLMs, with limited comparisons against established task-specific personalization models such as PENS, NAML, EBNR, LSTUR, MAMBA, SSD4Rec, GRU4Rec, and so on. This makes it difficult to determine whether PreMO\texttt{PreMO} is primarily a better preference interface for a generic LLM or whether it is a competitive benchmark against established specialized personalization models for downstream tasks.

  • The PersonaMovielens results are not convincing as evidence for query-relevant preference discovery. The target movie is retrospectively selected because the user later rated it 5 *, while the query is synthetically constructed using properties related to the target. In this setting, downstream success can entangle query-movie compatibility with history-conditioned personalization. More importantly, a later rating represents \textbf{\textit{post-consumption utility or satisfaction}}, not the true preference dimensions that caused the user to be interested in the movie. A movie may strongly match a user's prior preferences before watching but still receive a low rating because of the experienced quality of the acting, story, or other aspects that are not represented by the historical preference histories and records. Thus, this experiment is a weaker validation of the claim concerning recovery of the query-relevant preference state that should guide a current decision.

2. Dependence on the Frontier model, which is under-examined (Primary Blocker)

  • PreMO\texttt{PreMO} is substantially dependent on frontier-model supervision. Frontier models provide the initial memo/proposal demonstrations, contribute to the initial preference inventory and evaluation reference sets, and provide the semantic verification signals that determine proposer rewards and memo supervision.

  • The empirical evidence settings carefully ablate the structure of the verification path and study single-judge instability, so the issue is not that verifier quality is completely unexamined. However, these experiments do not establish whether the learned preference reasoning and downstream gains are robust to changing the actual teacher/verifier model or to a weaker or systematically different teacher. The alternative Gemini-built reference-pool experiment addresses robustness of evaluation ordering, but it does not test training-time teacher/verifier dependence because PreMO\texttt{PreMO} is not retrained under an alternative supervision model.

Questions for authors
  1. Any comparisons against specialized SOTA personalized benchmarks like MIND, Movielens, LAMP, and so on, relative to earlier stated models?

  2. Any clear ablation showing that pre-consumption interest is a weaker personalization setting than post-consumption rating? How can rating depend on user's prior history in any settings?

  3. How sensitive is PreMO\texttt{PreMO} training to the identity and quality of the frontier teacher/verifier?

  4. A direct correlation or stratified analysis between preference-reasoning metrics and downstream task success would help establish whether the improved intermediate reasoning is actually responsible for the downstream gains

9 comments
AuthorsOpenReview ↗

Response to Reviewer 6zSi (1/3): specialized personalization models

Thank you for the review. We ran new experiments for both primary blockers. This comment covers specialized models (Q1). Comment 2 covers PersonaMovieLens (Q2). Comment 3 covers the frontier teacher and verifier (Q3) and memo metrics (Q4).

W1a/Q1: Specialized personalization models

PREMO is a semantic preference interface for LLM decision-makers. Like other LLM-based personalization methods [1], it could complement ID-based recommenders. We trained four representative specialized models on our training splits and evaluated them on the same candidate sets as PREMO: GRU4Rec [2] and SASRec [3] (sequential recommenders), SVD++ [4] (collaborative filtering), and UniSAR [5] (joint search and recommendation with query input). GRU4Rec and SASRec also represent the listed sequential models MAMBA and SSD4Rec. PersonaMovieLens is built on MovieLens 32M (Appendix C.1). We therefore also trained these models on all 32M ratings, excluding the test users' later ratings. These ratings come from about 200k users, and our training split has 2,500. We skip SVD++ on APeB because it needs explicit ratings. The other listed models (NAML, LSTUR, EBNR, PENS) target news and take no query. We did not evaluate on MIND and LaMP. Neither has a query that states the user's current intent.

Method (Hit@1)Reads the queryAPeBPersonaMovieLensPersonaMovieLens, trained on all 32M ratings
Random choice–20.020.0–
GRU4Recno22.231.131.8
SASRecno22.639.045.4
SVD++non/a62.378.1
UniSARyes24.331.126.5
UniSAR, text item embeddingsyes25.343.646.4
Untrained 4B, query only (Table 1)yes55.162.4–
PREMO 4Byes65.973.7–
PREMO memo, Qwen3.5-27B as rankeryes71.081.1–

With the same training data, every ID-based model stays well below PREMO. SVD++ overtakes PREMO 4B only with all 32M ratings, and a larger LLM reading the same PREMO memo overtakes SVD++. On APeB, the ID-based models are close to random choice. On PersonaMovieLens, SVD++ trained on our training users only ties the untrained 4B that reads the query alone. PREMO itself is trained on our 2,500 training users only.

This is because our tasks depend on understanding the user's history and intent, and personalization data are often too sparse to learn item-ID embeddings. First, ID-based models learn from the relations between item IDs in many users' histories. Text embeddings help UniSAR, but it is still far below the untrained 4B that reads the query alone. Encoding item text does not provide this understanding. Second, APeB [6] has only about 6k sessions. In our APeB split, 29.1% of the test candidates never appear in the training histories, and 11.9% appear in only one history. MovieLens 32M is dense by comparison. With these ratings, SVD++ can learn good ID embeddings.

PREMO and ID-based models capture complementary signals, even when SVD++ is trained on all 32M ratings. ID-based models learn population-level patterns from many users. PREMO extracts the individual user's intent from their records and query. We compared the two on each PersonaMovieLens test instance: the PREMO memo with Qwen3.5-27B as ranker, and SVD++ trained on all 32M ratings (% of instances):

SVD++ rightSVD++ wrong
Memo-based policy right64.616.5
Memo-based policy wrong13.55.4

Each is right on more than 70% of the other's errors. An oracle that picks the right one for each instance would reach 94.6% Hit@1. The difference is largest on new items. On APeB, about 25% of the targets never appear in training. The memo-based policy ranks 67% of these targets first, and each ID-based model ranks 2.8% first. Bringing PREMO's memo into conventional recommenders (e.g., via KAR [1]) is a promising direction for future work.

[1] Towards open-world recommendation with knowledge augmentation from large language models, 2024.

[2] Session-based recommendations with recurrent neural networks, 2016.

[3] Self-attentive sequential recommendation, 2018.

[4] Factorization meets the neighborhood: A multifaceted collaborative filtering model, 2008.

[5] UniSAR: Modeling user transition behaviors between search and recommendation, 2024.

[6] APeB: Benchmarking personalization ability of large language model agents, 2026.

AuthorsOpenReview ↗

Response to Reviewer 6zSi (2/3): PersonaMovieLens and post-consumption ratings

W1b/Q2: Post-consumption ratings in PersonaMovieLens

We agree that post-consumption ratings reflect both initial consumption choice and subsequent satisfaction. We built PersonaMovieLens on MovieLens because it is a standard public dataset with long rating histories and with movie genres and tags. Since MovieLens logs ratings post-watching, PersonaMovieLens operationalizes personalized search as identifying the user's highest-satisfaction item (rating = 5.0) among candidate movies consumed post-cutoff. The negatives are lower-rated movies that also yield lower preference-match scores against the user's history. Modeling future ratings based on historical interactions remains a standard task in collaborative filtering [7]. GPT-5.5 writes the query from the target's genre family, without its title, year, tags, or synopsis (Appendix C.1). Amazon product-search benchmarks build queries from item categories in a similar way [8–10].

To address your concern, we vary how queries and candidates are built. One setting is close to choosing before watching (pre-consumption interest): all candidates match the query, but the user rated only the target. To isolate variables, we changed one component at a time. (1) Genre-only queries name only the target's high-level genre. (2) Unrated negatives are drawn either uniformly at random or from query-relevant movies. GPT-5.6 Luna judges this relevance from the query and movie metadata alone, without the user history.

PersonaMovieLens (Hit@1)Query onlyFull historyDPREMO
As in the paper62.464.972.873.7
Genre-only query47.450.966.769.7
Random negatives the user never rated82.683.885.486.5
Query-relevant negatives the user never rated43.845.151.853.6

When we change the query or the candidates, PREMO still leads D and the baselines. Its gain over the query alone is largest with the genre-only query. The gain also holds in the setting close to choosing before watching. The history thus helps predict both which movie the user chooses and how they rate it. Our other two tasks do not use ratings. The APeB target is a purchase, and PersonaMem-v3 scores replies.

We also gave PREMO another user's history. Its Hit@1 drops by 8.1 points (73.7 to 65.6). PREMO's gain therefore comes mainly from the user's own history.

[7] Collaborative filtering with temporal dynamics, 2009.

[8] Learning latent vector spaces for product search, 2016.

[9] Learning a hierarchical embedding model for personalized product search, 2017.

[10] A transformer-based embedding model for personalized product search, 2020.

AuthorsOpenReview ↗

Response to Reviewer 6zSi (3/3): frontier supervision and the metric–downstream link

W2/Q3: Sensitivity to the teacher and verifier

Our method is generally robust to the choice of teacher and verifier. In our current pipeline, GPT-5.4 generates the SFT warm-up data, the initial inventory, and the references E(x), and GPT-5.6 Luna serves as the RL verifier (Appendix C.10). We chose these models for cost and availability. At deployment, the 4B model writes the memo alone, with no frontier calls.

To test the sensitivity to the SFT warm-up with GPT-5.4, we ran an ablation without SFT warm-up. This run also replaces GPT-5.4's initial inventory with GPT-4.1 rubrics.

As shown below, even trained with RL alone (omitting SFT warm-up), PREMO already outperforms the teacher's memo across all tested downstream policies (e.g., +3.0 over GPT-5.4 memo when evaluated with GPT-5.4). SFT gives a better initialization (+3.8), but PREMO does not depend on SFT to surpass the teacher.

APeB Hit@1GPT-5.4 memoPREMO, RL onlyPREMO with SFT
GPT-4.167.168.670.3
GPT-5.466.669.673.4
GPT-5.569.470.472.4

We further ablate the verifier with Qwen3.8-27B.

With an open model as verifier, PREMO performs as well as with GPT-5.6 Luna. We retrained PREMO on PersonaMem-v3 for one round with the open model Qwen3.8-27B as verifier and memo judge. It runs on four H100s. The teacher and the evaluation stay the same.

PersonaMem-v3, after one roundCov.Faith.Ground.TaskTraining cost
GPT-5.6 Luna as verifier and judge (paper's run)74.783.764.382.180–88 H100-hours, USD 110–133
Qwen3.8-27B as verifier and judge75.087.165.081.5about 130 H100-hours, no API

Open-model verification matches or exceeds frontier verification on all intrinsic memo metrics, with comparable downstream performance and no API cost. The improvements of PREMO thus come mainly from the method's design, which does not rely on a specific frontier model. Each ablation removes one frontier dependence. RL-only training uses neither the SFT data nor the initial inventory of GPT-5.4, and the open verifier replaces GPT-5.6 Luna.

Q4: Do memo metrics explain downstream success?

We would like to note that high-quality memos are important to the success of downstream outcomes. The improved memo quality of PREMO can effectively bring downstream improvements. For each task, we sampled 500 memos from PREMO's checkpoints after each round, evenly across bins of the score being tested. We then computed the Pearson correlation between the score and the instance's downstream outcome (Hit@1, or the task score on PersonaMem-v3). We tested coverage alone and an aggregate score, 0.5 × coverage + 0.25 × faithfulness + 0.25 × grounding:

Pearson r with downstream successCoverageAggregate score
APeB0.2490.215
PersonaMem-v30.1350.173
PersonaMovieLens0.2190.265

All correlations are positive. How much personalization can help differs across queries and tasks [11].

We hope these results resolve the main questions and are glad to provide further details.

[11] To personalize or not to personalize: Modeling queries with variation in user intent, 2008.

Reviewer_rVxxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of personalized decision-making, specifically, how to identify user preferences relevant to a query from a long and noisy history. The paper proposes PREMO, which summarizes relevant user preferences in a memo while remaining faithful to historical evidence. PREMO uses a single model that alternately assumes two complementary roles: as a proposer, it discovers new preferences; and as a memo policy, it generates memos based on the query and historical records. Experiments show that PREMO has made some progress.

Critical strengths
  • This paper provides a wealth of theoretical analysis, offering strong support for the proposed technology and enhancing its interpretability.
  • Although the training process is complex, PREMO’s inference during the deployment phase remains simple and efficient, enhancing its practicality.
Critical weaknesses
  • The necessity of some technologies is still unclear or has not yet been verified.
  • Some of the experimental results are insufficient to support the conclusions claimed by the authors.
Questions for authors
  1. In Section 3.3, verified preferences and invalid claims yield rewards, but claims outside both observed sets receive no atom-level score. The authors should provide the frequency of such claims—which are not covered by the supervision—during the actual training process.
  2. In Theorem 2, the authors simplify Recognition and Expression into two scalars, u and v, and define the memo gain g as uv. All subsequent proofs regarding complementary training are based on this overly simplified mathematical model and lack sufficient persuasiveness. Unless experiments or other ways can demonstrate the validity of this modeling approach, the reliability of the theoretical analysis will be significantly compromised.
  3. The author should first conduct full proposer training, followed by full memo training, and then compare the results of this variant with those of PREMO. This would demonstrate whether repeated alternating training is truly necessary.
  4. Training PREMO is very expensive, but at the same time, Table 1 does not show sufficiently good results; the authors need to provide further explanation for this.
  5. In addition, it is necessary to compare the differences in efficiency between the baselines and PREMO during training and deployment in order to better evaluate the advantages of the proposed method.
  6. Table 3 shows that, in the final training round, training the Memo Policy one last time actually reduced the model’s performance on some metrics. This suggests that the complementary objectives claimed in the paper are, in fact, somewhat conflicting and require further explanation.
6 comments
AuthorsOpenReview ↗

Response to Reviewer rVxx (1/3): unscored claims and the theoretical model

Thank you for the review. We ran a new experiment or analysis for every question. Q1–Q3 address W1, whether each component is needed and verified: unscored claims (Q1), the two-scalar model (Q2), and repeated alternation (Q3). Q4–Q6 address W2, whether the results support our claims: the size of the gains (Q4), efficiency (Q5), and the final memo phase (Q6). Comment 2 answers Q3 and Q6, and Comment 3 answers Q5 and Q4.

Q1: How often do training memos contain claims outside Ik∪S^k−I_k \cup \hat{S}_k^-?

About half of the claims fall outside both sets, already at the starting checkpoint, because a memo is free text while the sets list atomic preferences. The proposer states one atom per slot. A memo is a paragraph instead, and an LLM often packs several attributes into one phrase. Splitting "gentle, practical, budget-friendly hair care" (an APeB memo, Table 11) gives three claims. If the inventory has an atom only for gentle hair care, the other two fall outside both sets even when the history supports them. These claims leave the memo reward unchanged: like rubric-based LLM judges [1], it scores only the atoms in the two sets, crediting verified atoms (IkI_k) and penalizing rejected ones (S^k−\hat{S}_k^-; Eq. 5).

What matters is whether these claims are supported, and the proposer role supervises exactly that. It is steered toward preferences not yet in the inventory, and verification scores each proposal: +1 for a new verified atom, −0.5 for a rejected one, and −1 for citing a missing record (Eq. 4). Verified proposals join IkI_k and rejected ones join S^k−\hat{S}_k^-, so the next memo phase scores them. GPT-5.6 Luna split 1,000 training memos per checkpoint on PersonaMem-v3 into claims and labeled each unmatched claim as supported by the history or not:

Route, start → endOutside both sets (% of claims)Outside both sets and unsupported (% of claims)
D45.0 → 55.29.3 → 12.2
A (PREMO)45.0 → 50.09.3 → 7.3

Most of the claims outside both sets are supported by the history. PREMO reduces the unsupported ones, while memo-only training, which lacks the proposer's supervision, increases them. Both routes train on the same inventories, so the difference is not due to D having fewer scored atoms. The evaluation also checks the memo against the history, so these claims are already taken into account (grounding and faithfulness, Appendix C.2).

Q2: Is the two-scalar model of Theorem 2 justified?

Theorem 2 models the two abilities a memo needs, recognition and expression, as two scalars uu and vv, with memo gain g=uvg = uv. The model is simplified, but each of its steps matches what we observe in training. It follows a standard model of slow learning from small initialization [2]. To test it, we measured recognition directly, as Section 2 defines it. pθ(r∣x)p_\theta(r \mid x) is the probability that a checkpoint, given (Q,H)(Q, H) and one atom rr written as a rubric question with its answer, says the history supports this answer. Unlike Pass (Table 3), every checkpoint judges the same atoms. On the PersonaMem-v3 test set, the atoms of E(x)E(x) serve as valid atoms, and atoms rejected by verification as invalid ones:

PersonaMem-v3 checkpointpθp_\theta, validGap: valid − invalid
Base79.8−1.1
Warm-up (start of A and D)98.04.7
One proposer phase after the warm-up96.210.0
D99.21.8
A (PREMO)98.27.7

Because invalid atoms are drawn from the same user history, raw acceptance probability pθ(valid)p_\theta(\text{valid}) may be confounded by history-level priors. Recognition therefore shows in the gap, which is near zero for the base model. The empirical trajectory directly corroborates the analysis.

  • The memo improves only when recognition and expression improve together (g=uvg = uv). Table 13 tests this on APeB. Starting the memo phase from the proposer checkpoint improves recognition. Adding the new atoms gives the memo phase more preferences to express. Each change alone raises Hit@1 by at most 0.4 points; together they raise it by 2.2.

  • Memo-only training barely raises weak recognition (u˙=v(1−uv)\dot u = v(1-uv)). D lowers the gap from 4.7 to 1.8.

  • The proposer role raises recognition directly (u˙=1−u\dot u = 1-u). One proposer phase raises the gap from 4.7 to 10.0.

  • The next memo phase starts from a larger uu. A ends with a larger gap than D (7.7 vs. 1.8) and higher coverage (82.9 vs. 74.9; Table 1).

Theorem A.4's first bound also holds without the scalar model. It compares actual GRPO updates of the two routes. Where the model is used, Appendix A.5 carries its approximation error, and Appendix A.3 relaxes its assumptions.

[1] Rubrics as rewards: Reinforcement learning beyond verifiable domains, 2025.

[2] Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2014.

AuthorsOpenReview ↗

Response to Reviewer rVxx (2/3): repeated alternation and the final memo phase

Q3: Is repeated alternation necessary?

Yes. The two roles share one model, and long training on one role weakens the other, which our theory describes as a task-switch gap (Appendix B.4). RL fine-tuning of an LLM on one objective can lower its performance on other tasks, and mixing in updates on its original objective reduces this loss [3]. In PREMO, a proposer phase improves the shared features but shifts the memo readout, which the next memo phase must readapt. On APeB, one 300-update proposer phase lowers memo coverage from 72.1 to 64.6 (θP\theta_P; Tables 3 and 13). Short phases keep this gap small enough for the next memo phase to recover. A single long proposer stage can widen it beyond what one memo stage recovers.

We ran the sequential variant of Q3, all proposer training followed by all memo training, and it ends with the lowest grounding and task score. On PersonaMem-v3 at 4B, the variant first runs all proposer training: 600 or 1,200 updates, with verification every 300 updates so that the inventory grows as in PREMO. It then runs all memo training, 400 or 800 updates, on the final inventory. We compare it with D and PREMO after the same number of updates, from the paper's runs after two and four rounds (four rounds: Table 1).

PersonaMem-v3, 4BCov.Faith.Ground.Task
Two rounds: D75.083.762.780.6
Two rounds: sequential77.981.158.978.8
Two rounds: PREMO80.685.364.182.3
Four rounds: D74.984.062.981.2
Four rounds: sequential78.986.958.578.9
Four rounds: PREMO82.989.565.583.3

At both budgets, PREMO is best on every metric, while the sequential variant's task score barely changes with the larger budget. Its memos cover more than D's but are less grounded. We infer that one memo stage cannot close the gap left by a long proposer stage. We will include a detailed analysis in the final version.

Q6: Why does the final memo phase lower faithfulness relative to θP\theta_P?

The drop in faithfulness during the final memo phase reflects an expected, controlled precision–recall trade-off between orthogonal objectives: The proposer reward penalizes rejected atoms and citations of missing records, and gives zero for an empty slot (Eq. 4). After a proposer phase, the model therefore writes fewer and safer claims (θP\theta_P). The memo reward pays for coverage, so the next memo phase adds claims, from 10.1 to 13.0 per memo on APeB. Coverage and Hit@1 rise, while faithfulness and grounding drop slightly (Table 3). D writes even more claims (14.1) but covers less than PREMO.

Each phase trains one factor of Theorem 2: the proposer phase raises recognition and thus faithfulness, and the memo phase raises expression and thus coverage. Alternating them, as in Q3, balances the two. To show this, we compared faithfulness at equal coverage. We evaluated intermediate checkpoints of D and of PREMO's final memo phase on APeB every 40 updates and interpolated linearly:

CoverageFaith. of PREMO's final memo phaseFaith. of D
64.6 (θP\theta_P)80.773.5
70.8 (D's endpoint)78.971.6
73.8 (PREMO's endpoint)77.1not reached

PREMO reaches a better trade-off than D (a higher plateau): at the same coverage, it is about 7 points more faithful, and D never reaches PREMO's coverage. With the same number of updates, PREMO ends above D on every metric (Table 3). D gains coverage but becomes less faithful than the base model. PREMO gains coverage and becomes more faithful than the base model. On all three tasks, the memos that are more faithful than PREMO's all cover less (Table 1).

[3] Training language models to follow instructions with human feedback, 2022.

AuthorsOpenReview ↗

Response to Reviewer rVxx (3/3): cost and the size of the gains

Q5: How efficient is PREMO in training and deployment?

PREMO is efficient in both training and deployment: one training round already beats every baseline below, including D after four rounds, and the memo shortens the downstream input from 10,763 to 447 tokens. Offline training is a one-time amortized cost, and PREMO can be trained with flexible budgets: four rounds, one round, or one round with an open verifier. The table below compares costs on PersonaMem-v3 at 4B. Default training is for four rounds. Table 17 (Appendix C.13) gives the cost of one PREMO round, and we estimate D's cost from its cost per memo update. Downstream times are per query on one H100, and Task is the PersonaMem-v3 score of Table 1:

MethodTraining H100-hoursTraining API costDownstream input tokens / seconds per queryTask
PREMO320–352USD 440–532447 / 2.5183.3
PREMO, one round80–88USD 110–133similar to PREMO82.1
PREMO, one round, Qwen3.8-27B as verifier and judgeabout 130nonesimilar to PREMO81.5
Dabout 307USD 440–530similar to PREMO81.2
P-Check22about USD 91386 / 2.6673.8
Mem0nonenone538 / 1.9375.3
Full historynonenone10,763 / 4.2163.5

Every PREMO variant beats P-Check, Mem0, and the full history by a wide margin. PREMO's training costs more than P-Check's, but it is a one-time cost. Even one round already exceeds D after four rounds (82.1 vs. 81.2).

With an open verifier, PREMO needs no API payment and still works well. We replaced GPT-5.6 Luna with Qwen3.8-27B on four H100s. One round then takes about 130 H100-hours and no API calls, and it reaches 81.5.

At deployment, the memo shortens the downstream input and the time per query. The downstream policy reads 447 tokens instead of 10,763 and takes 2.51 s instead of 4.21 s per query. Writing the memo takes one 4B call (2.37 s). P-Check and Mem0 also need such a call to prepare their input. The memo call does not depend on the candidates and can run while they are retrieved. The shorter input saves most when the downstream policy is larger or billed per token, as with the frontier policies of Table 7.

Q4: Are the results in Table 1 strong enough for the training cost?

PREMO learns a short, reusable representation of the user's history. It raises the plateau that downstream policies reach, at a cost that can be lowered (Q5). Like other learned representations [4], the memo can be used directly by different downstream policies (Table 7) or used to fine-tune a downstream policy for a specific task. Table 1 tests the memo with an untrained 4B downstream policy, so its scores reflect the memo alone; even so, PREMO is best on all three tasks. On APeB, we additionally trained the downstream policy by GRPO with the ranking outcome as the reward, at matched updates, reading the full history or one of three memos:

APeB Hit@1, input to the downstream policyUntrained (Table 1)Trained on the outcome
Full history54.956.7
Base memo57.863.4
GPT-5.4 memo61.567.8
PREMO memo65.971.0

Downstream training works much better on a memo than on the full history. On the full history it gains only 1.8 points, because a 0/1 outcome over a long history is too sparse to learn from.

We hope these new results address your concerns and are glad to provide further information.

[4] Representation learning: A review and new perspectives, 2013.

Reviewer_LT2EOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

PREMO trains one language model in two alternating roles: a proposer identifies query-relevant preferences in a long user history, and a memo writer compresses supported preferences into a short memo for a downstream decision-maker. A multi-step verifier admits or rejects proposed preference atoms, supplying rewards to both roles. On shopping, movie, and personalized-response tasks, the resulting memos generally outperform raw-history and summary baselines.

Critical strengths
  1. The direct-memo route starts from the same warm-up checkpoint, uses the same inventories, and has the same number of GRPO updates (Table 3; Appendix C.12). The proposer improves recognition and helps recover faithfulness lost under memo-only training; a checkpoint-by-inventory experiment further probes the handoff (Table 13). This supports the paper's main mechanism better than an SFT-only comparison would.

  2. Ablating verification stages reduces both memo quality and APeB accuracy (Table 4). The authors report human audits of reference atoms and judge outputs, an alternative reference pool, and user-level bootstrap intervals (Appendix C.3–C.4). On APeB, the PREMO memo improves Hit@1 over the GPT-5.4 memo across five downstream policies (Table 7).

Critical weaknesses
  1. GPT-5.4 proposes and verifies the test atoms over four rounds, capped at 20 per instance (Appendix C.2). The human audit assesses support for included atoms (91% accepted), but does not estimate how many relevant preferences the pool misses. Thus higher coverage establishes better recovery of this pool, not necessarily more complete or correct personalization. The authors' alternative-pool and pairwise audits reduce, but do not fully remove, this limitation.

  2. Relative to matched memo-only training, PREMO's improvement has a bootstrap interval excluding zero on APeB (+3.7 Hit@1) and PersonaMem-v3 (+2.1 task-score points), but not on PersonaMovieLens (+0.9, interval spanning zero; Appendix C.4). APeB also retains only the top 30% of sessions by query–purchase fit, and PersonaMem-v3 uses query-aware compacted histories rather than the original approximately 400k-token logs (Appendix C.1). Testing unfiltered and less curated histories would better establish whether the mechanism works under the claimed long, noisy conditions.

Questions for authors
  1. On a human-audited sample, what fraction of query-relevant, history-supported preferences are absent from the GPT-5.4 reference pool? Does the relative coverage ranking hold when those missing preferences are included?

  2. How do the matched-update improvements change on the unfiltered APeB sessions and on PersonaMem-v3 with a less query-dependent history selection? These results would help establish whether the gains generalize beyond the curated settings.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LT2E: preferences missing from E(x) and gains outside the curated settings

Thank you for the review.

W1/Q1: Preferences missing from E(x)

We agree that the C.3 audit checks the atoms in E(x) but not what E(x) misses. However, in personalized systems, the full set of a user's preferences cannot be listed exhaustively [1,2]. Even for human annotators, it is challenging to reliably list every preference supported by a history of about 120 records. The preferences can also vary across different granularities (e.g., "prefers black" vs. "prefers black quick-dry running tops").

Instead of directly measuring the recall, we can still check the false negatives and establish a lower bound on what E(x) misses by having annotators re-check the atoms that our pipeline proposed but rejected. While building E(x), the verification path rejected some atoms that GPT-5.4 proposed (Appendix C.2); these are candidate false negatives. On the 100 APeB test users of the C.3 audit, two annotators independently marked the rejected atoms supported by the history and relevant to the query, and added any other such preference; a preference was kept only if the other annotator agreed. For example, for a user buying contact lenses, E(x) covers six attributes, such as diameter and base curve; the annotators added the degree of astigmatism correction and whether a lens case and tools are included. At least one annotator flagged a missing preference in 31% of the instances, but after cross-checking and de-duplication only 13% keep one: 16 preferences, about 2.3% of the atoms in E(x) on these users. The 20-atom cap is not the cause: on the APeB test set, E(x) averages 6.6 atoms, and no instance reaches the cap.

E(x) thus misses few preferences that annotators agree on, and on these PREMO still covers the most. We scored the memos of the 100 users on the 16 missed preferences with the same judge:

Memo (100 audited users)Cov. on E(x)Cov. on the 16 missed preferences
PREMO72.656.3
D71.053.1
GPT-5.4 memo68.453.1
SFT (GPT-5.4)62.240.6
Mem041.331.3
P-Check20.718.8

All methods score lower on these preferences than on E(x), but PREMO still leads, so its higher coverage is not limited to the reference pool. We will include a more detailed analysis in the final version.

W2/Q2: Gains over D outside the curated settings

The paper filters APeB sessions because vague queries may leave the reference set incomplete, and compacts PersonaMem-v3 histories because they exceed the model's context. PREMO's lead over D holds without these two steps. The APeB filter keeps the 30% of sessions whose purchase best matches the query and never shortens the history. For vague queries (e.g., "outfit"), annotators agreed that E(x) misses relevant preferences in 47% of 100 removed sessions, against 13% of kept ones. PersonaMem-v3 histories average about 400k tokens, more than Qwen3.5-4B's native 262k-token context, so the paper keeps up to 120 records: privacy boundaries and corrections first, then those most relevant to the query (Appendix C.1).

We removed each step at evaluation, without retraining. On APeB, we evaluated the removed sessions of users outside the training and validation splits. On PersonaMem-v3, we used the most recent records instead of the query-relevant ones, and 240 records instead of 120. On PersonaMovieLens, we changed the query or the ranker. Intervals follow Appendix C.4.

DatasetSettingDPREMOPREMO − D [95% CI]
APeB (Hit@1)Kept sessions (paper)62.265.9+3.7 [1.1, 6.1]
Removed sessions60.663.3+2.7 [0.6, 4.7]
PersonaMem-v3 (Task score)120 query-relevant records (paper)81.283.3+2.1 [0.2, 4.4]
120 most recent records78.581.3+2.8 [0.5, 5.2]
240 query-relevant records80.883.4+2.6 [0.1, 5.4]
240 most recent records80.082.7+2.7 [0.3, 5.1]
PersonaMovieLens (Hit@1)As in the paper72.873.7+0.9 [−0.8, 3.0]
Genre-only query66.769.7+3.0 [0.2, 5.7]
Qwen3.8-27B as ranker79.783.1+3.5 [1.2, 5.7]

In all six new settings, PREMO leads D with a 95% interval above zero.

On PersonaMovieLens, the long histories make model capacity the main bottleneck. The model reads the longest histories on this task (18.0k tokens; Table 5). A larger model can raise PREMO's results, as the 35B model does on APeB (Table 3). With the same 4B memos, PREMO's lead over D becomes clear when the query is less specific (+3.0) or a stronger model uses the memo to rank the candidates (+3.5).

We hope this addresses both concerns and are glad to provide further details.

[1] Collaborative filtering for implicit feedback datasets, 2008.

[2] Truth is a lie: Crowd truth and the seven myths of human annotation, 2015.

AuthorsOpenReview ↗

Follow-up on our response

Thank you again for your review. As the discussion period nears its end, we would like to check whether our response has addressed your concerns about the preferences missing from E(x) and the gains outside the curated settings. If you have any follow-up questions, we would be glad to clarify them quickly.

Thank you for running the additional experiments. The new results address my concerns about data filtering and history selection, and the audit of missing preferences provides useful additional evidence. I will maintain my positive recommendation.

Thank you for the constructive discussion and for your positive recommendation. We will include the new results in the revision and discuss the remaining limitations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

thank all reviewers for the discussion, summary of the revision

We thank all reviewers for the discussion. We have uploaded a revised PDF in which all changes are in blue.

Main text

  • Section 2 and RQ2 (Reviewer rVxx). Theorem 2 is presented as a scalar model of recognition and expression. Table R1 reports our estimate of recognition at training checkpoints: proposer training raises it, while memo-only training lowers it. RQ2 adds a sequential schedule, which ends below alternation in grounding and task score (Figure R1 and Table R2). With Tables 3, 4, and 13, these results show what each component of PREMO adds.
  • Setup (Reviewer LT2E and 6zSi). PersonaMovieLens is described as choosing the movie the user will enjoy most after watching. The teacher and verifier are presented as the configurations of the main results, with two alternatives in Appendix C.15.
  • Related Work (Reviewer 6zSi). A new paragraph covers search and recommendation models and points to the comparison in Appendix D.
  • Limitations (Reviewer LT2E, 6zSi and rVxx). The Conclusion refers to the limitations in Appendix E.

Appendix

  • C.3 (Reviewers LT2E and 6zSi): preferences missing from the reference pool (Table R3), and correlations between memo metrics and downstream success (Table R4).
  • C.8 (Reviewer rVxx): details of the recognition estimate, and the sequential schedule after two and four rounds.
  • C.14 (Reviewers LT2E and 6zSi): robustness to data construction: the APeB sessions removed by the filter, other PersonaMem-v3 record selections, and PersonaMovieLens variants.
  • C.15 (Reviewer 6zSi): training without the GPT-5.4 warm-up, and training with an open verifier.
  • Appendix D (Reviewer 6zSi): specialized search and recommendation models, matched information, memos written before the query is known, and combinations of PREMO with SVD++. D.3 adds a check of memos written without the query against the user's history, judged by GPT-5.6 Luna and checked by hand on a sample. Almost all of these memos state a relevant preference that the history supports, but most miss details that the memo written with the query captures.

We hope the revision makes these results easy to find, and we are glad to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.