acceptodds
Under review as a conference paper at ICLR 2027

PULSE: Provenance-Aware User-Profile Learning and Selective Evaluation for Long-Term LLM Personalization

Abstract

Long-term LLM personalization aims to adapt an assistant’s responses to user preferences as they evolve across interactions. Existing approaches typically represent users through latent embeddings, natural-language profiles, or learned summaries to guide future response generation based on interaction history. However, profile updating and utilization often remain implicit, leading to two challenges: situational or uninformative feedback may be incorporated into persistent preferences, while valid but context-irrelevant profile information may distort the evaluation of responses to the current request. Long-term personalization therefore requires determining when recent interactions provide reusable preference evidence that justifies profile revision and when stored information should influence response selection. To address these challenges, we propose PULSE, a provenance-aware framework that explicitly models both profile updating from interaction evidence and selective profile utilization in personalized response evaluation. PULSE selects interaction events containing reusable preference evidence, determines the appropriate update state (no update, local preference refinement, or long-term preference revision), and generates evidence-grounded profile statements. It then converts updated profiles into request-specific criteria and evaluates candidate responses through a criterion-guided personalized residual reward. This residual reward selectively corrects a profile-free anchor reward when the profile provides applicable guidance, preserving general response quality while preventing irrelevant preferences from distorting response selection. Experiments on HorizonBench with both static- and evolving-preference instances show that PULSE achieves effective personalized ranking and preference-change discrimination while maintaining evidence-grounded profiles and limiting the influence of unsupported or context-irrelevant profile information.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Personalization, Dynamic User Profiling, Personalized Reward Modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zezeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes PULSE, a framework for long-term LLM personalization that combines evidence-grounded profile updating with selective preference utilization. It introduces tri-state profile updates and criterion-guided residual rewards to improve personalized response ranking.

Critical strengths
  1. This work addresses two relevant challenges in long-term LLM personalization: determining when user profiles should be updated and when stored preferences should influence response evaluation.

  2. PULSE combines evidence-grounded profile updating with request-specific criterion construction and residual reward modeling. This structured design improves the traceability of profile revisions.

  3. The experiments cover both static and evolving preferences, seen and unseen users, and two backbone models. The results show consistent improvements in overall response-ranking performance.

Critical weaknesses
  1. The central claim of selective personalization is not sufficiently isolated from ordinary instruction following. In the case study, the user explicitly requests a "choose-your-own-adventure" format, yet PULSE derives essentially the same criterion from the user profile. The resulting ranking improvement therefore does not clearly demonstrate the additional value of historical preferences. A query-only criterion baseline is needed to distinguish genuine personalization gains from improved evaluation of explicit request requirements.

  2. The experimental evaluation is limited to offline candidate ranking on a single benchmark, without demonstrating improvements in open-ended response generation or long-term interactive personalization. Moreover, comparisons with closely related criterion-based reward modeling approaches are missing.

  3. The provenance-aware design does not fully guarantee that profile updates are semantically supported by historical evidence. The training objective primarily checks the structural consistency of evidence references rather than whether the cited interactions actually entail the inferred preferences. Unsupported preference interpretations may still be incorporated into the profile despite having valid citations.

  4. The tri-state update mechanism assigns a single update state to the entire profile, potentially overlooking situations where different preferences require different update decisions. Furthermore, the reported update-state Macro-F1 is only 54.3%, and the evaluation includes just six reference cases of long-term revision.

  5. The profile updater is optimized using downstream utility computed by the learned personalized reward model, creating a risk of reinforcing existing reward-model biases rather than correcting inaccurate user profiles. Although the authors introduce evidence-consistency constraints, they do not sufficiently investigate this potential self-reinforcement or the stability of repeated profile updates.

Questions for authors

Pls see the weakness

3 comments

Q1: The central claim of selective personalization is not sufficiently isolated from ordinary instruction following. In the case study, the user explicitly requests a "choose-your-own-adventure" format, yet PULSE derives essentially the same criterion from the user profile. The resulting ranking improvement therefore does not clearly demonstrate the additional value of historical preferences. A query-only criterion baseline is needed to distinguish genuine personalization gains from improved evaluation of explicit request requirements.

A1: We thank the reviewer for highlighting the importance of distinguishing historical personalization from explicit instruction following. We clarify the case study and provide the requested Query-only baseline.

(1) Clarification of the case study. Figure 4 primarily illustrates how PULSE updates an outdated preference for numbered options to reflect the user's newly expressed preference for branching choices, while preserving the supporting evidence. To directly assess the additional contribution of historical preferences beyond explicit request requirements, we conduct the following controlled experiment.

(2) Additional Query-only criterion baseline. We introduce a Query-only baseline that constructs evaluation criteria exclusively from the current request, without accessing historical interactions or user profiles. It uses the same frozen anchor, candidate responses, and evaluation protocol as PULSE, thereby isolating the contribution of history-grounded personalized criteria.

Backbone / SettingQuery-only Top-1PULSE Top-1Improvement
Qwen3-4B / Seen90.14%90.52%+0.38 pp
Qwen3-4B / Unseen89.43%89.95%+0.52 pp
Llama-3.2-3B / Seen90.73%92.68%+1.95 pp
Llama-3.2-3B / Unseen90.46%90.83%+0.37 pp

PULSE consistently outperforms Query-only across all four settings, with Top-1 improvements ranging from 0.37 to 1.95 percentage points. Since Query-only already captures requirements explicitly stated in the current request, these results provide direct evidence that incorporating historical preferences yields additional personalization benefits beyond ordinary instruction following.

Q2: The experimental evaluation is limited to offline candidate ranking on a single benchmark, without demonstrating improvements in open-ended response generation or long-term interactive personalization. Moreover, comparisons with closely related criterion-based reward modeling approaches are missing.

A2: We thank the reviewer for these valuable suggestions. We clarify the scope of our long-horizon evaluation and the differences from closely related criterion-based methods.

(1) Long-horizon personalization and open-ended generation. PULSE is designed to improve evidence-grounded profile updating and personalized response evaluation, rather than directly modifying the response-generation policy. As formulated in Section 2, its objective is to select the preferred response from a candidate set using the updated user profile. Therefore, Top-1 accuracy and MRR directly measure the effectiveness of the proposed personalization mechanism. Evaluating methods on identical candidate responses also isolates personalization gains from differences in response generation.

Regarding long-horizon personalization, HorizonBench provides chronological, multi-session histories covering both static and evolving preferences. Crucially, PULSE updates user profiles autoregressively using its own previous predictions, without reference-profile teacher forcing. Consequently, subsequent response evaluations depend on earlier profile updates, allowing the experiments to assess personalized response selection throughout evolving interaction histories.

(2) Additional comparisons for criterion-based personalized evaluation.

To further examine the effectiveness of criterion-based personalization, we complement the existing w/o Criterion-Guided Reward ablation in RQ2 with two additional controlled baselines: Anchor-only, which removes the personalized residual, and Query-only, which constructs evaluation criteria solely from the current request without accessing historical preferences. Both use the same frozen anchor and candidate responses as PULSE.

Backbone / SettingAnchor-onlyQuery-onlyPULSE
Qwen3-4B / Seen90.12%90.14%90.52%
Qwen3-4B / Unseen88.14%89.43%89.95%
Llama-3.2-3B / Seen90.52%90.73%92.68%
Llama-3.2-3B / Unseen88.92%90.46%90.83%

PULSE consistently outperforms both baselines across all four settings. Together with the RQ2 ablation, these results demonstrate the benefits of criterion-guided personalized evaluation, historical preference information beyond the current request, and residual correction beyond the frozen anchor.

Q3: The provenance-aware design does not fully guarantee that profile updates are semantically supported by historical evidence. The training objective primarily checks the structural consistency of evidence references rather than whether the cited interactions actually entail the inferred preferences. Unsupported preference interpretations may still be incorporated into the profile despite having valid citations.

A3: We thank the reviewer for this important concern. We would like to clarify that PULSE distinguishes structural provenance constraints from semantic evidence support.

(1) The update objective goes beyond structural citation consistency. The attribution term (A(Z)A(Z)) enforces traceable evidence references, but it is only one component of the complete update objective. As defined in Eq. (4), the updater additionally optimizes current and future preference-ranking gains, preference-drift consistency, and profile-growth regularization. Therefore, candidate update traces are evaluated not merely by the validity of their citations, but also by their downstream utility on labeled preference comparisons.

(2) Semantic evidence support is explicitly evaluated. RQ3 introduces Evidence Support Rate (ESR), which directly assesses whether newly introduced or revised profile statements are semantically supported by their cited historical interactions, rather than merely containing valid evidence IDs. As shown in Table 2, PULSE achieves 82.2% ESR, compared with 72.4% for Prompted Tri-State, representing a 9.8-percentage-point improvement.

These results show that PULSE does not rely solely on structural citation validity: its profile updates are additionally guided by downstream preference utility, while their semantic evidence support is independently assessed through ESR. The observed improvement provides empirical evidence of more reliable evidence-grounded profile updates.

Q4: The tri-state update mechanism assigns a single update state to the entire profile, potentially overlooking situations where different preferences require different update decisions. Furthermore, the reported update-state Macro-F1 is only 54.3%, and the evaluation includes just six reference cases of long-term revision.

A4: We thank the reviewer for raising these concerns. We clarify the distinct roles of update-state prediction and evidence-grounded profile generation, and provide additional human-validation results.

(1) Why a single update state is meaningful. The tri-state label specifies the scope of revision justified by the recent evidence, rather than assigning the same edit operation to every profile statement. Specifically, No Update preserves the existing profile; Local Refinement incorporates context-bound preferences without replacing established long-term preferences; and Long-term Revision permits changes to persistent preferences when supported by evidence.

Importantly, the update state determines what kind of revision is warranted, while evidence-grounded generation determines which statements to change. Under Long-term Revision, for example, the generator may revise an outdated persistent preference while retaining unrelated preferences and their provenance. Thus, a single update state does not imply uniform changes across the profile. The distinction between Local Refinement and Long-term Revision remains essential because it explicitly controls whether established preferences can be revised.

(2) Update-state performance and human validation. On the original LLM-adjudicated references, PULSE achieves 54.3% Macro-F1, outperforming Prompted Tri-State (40.8%) by 13.5 percentage points. To further examine the limited number of long-term-revision references, we conducted additional human validation on the same 175 instances:

ReferenceNo UpdateLocal RefinementLong-term RevisionPULSE Macro-F1
LLM-adjudicated52117654.30%
Human-validated66733657.61%

Human validation identifies 36 long-term-revision cases, compared with only six in the original LLM-adjudicated references. Under these human-validated labels, PULSE achieves 57.61% Macro-F1.

Together, these results clarify the functional role of tri-state attribution and provide additional human-validated evidence for its effectiveness.

Q5: The profile updater is optimized using downstream utility computed by the learned personalized reward model, creating a risk of reinforcing existing reward-model biases rather than correcting inaccurate user profiles. Although the authors introduce evidence-consistency constraints, they do not sufficiently investigate this potential self-reinforcement or the stability of repeated profile updates.

A5: We thank the reviewer for raising these concerns. We clarify two aspects of PULSE's training and evaluation.

(1) Reward-model self-reinforcement. PULSE's updater is not optimized by maximizing reward scores on its own generated responses. Instead, candidate update traces are evaluated against dataset-provided preference labels. Specifically, Eqs. (4–8) measure whether an updated profile improves the ranking margin between labeled preferred and rejected responses, relative to the frozen profile-free anchor, on both the current request and subsequent requests. For preference-drift cases, the objective additionally measures whether the updated profile favors the current preference over the outdated-preference distractor. Furthermore, the criterion inducer and reward model remain fixed during updater optimization. Thus, update supervision is tied to externally provided preference comparisons, rather than treating the reward model's own predictions as ground-truth preference labels.

(2) Stability under repeated updates. Our evaluation does not assume oracle profiles at each interaction. As described in Section 4.1, PULSE updates profiles autoregressively using its own previous predictions, without reference-profile teacher forcing. Consequently, the results in Table 1 evaluate downstream personalization using profiles generated through successive updates, rather than reference-profile teacher forcing. The results in Table 1, covering both static and evolving preferences, therefore evaluate personalization under successive model-generated profile revisions. Additionally, the explicit No Update candidate and profile-growth regularization discourage unnecessary changes during training.

Reviewer_FKFFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats long-term LLM personalization as two separate decisions: (1)when recent feedback justifies revising a user profile, and (2) which stored preferences should affect the evaluation of the current response. PULSE handles the first with a factorized updater. It selects evidence interactions, assigns an update state (no update, local refinement, or long-term revision), and writes a profile whose new statements cite their supporting evidence. The updater is trained with DPO on self-generated traces scored by downstream utility, so it needs no manual state labels. For the second decision, PULSE turns the applicable profile statements into request-specific criteria and uses them to train a residual reward on top of a frozen, profile-free anchor reward.

On HorizonBench, PULSE achieves the best overall Top-1 accuracy and MRR against strong baselines (RAG, SFT, DPO, VPL, PAL, PLUS), with consistent results across two backbones and both seen and unseen users. The work offers an interpretable, traceable path from user feedback to personalized response selection, which is valuable for building trustworthy long-term assistants.

Critical strengths

Separating "should the profile change?" from "should this preference apply here?" captures two real failure modes in fig1. The provenance design, in which every new statement cites evidence and every criterion cites a source statement, makes the system more inspectable compared with latent-variable personalized reward models.

Critical weaknesses

It is not yet clear how much of the improvement comes from the two proposed mechanisms rather than from the trained anchor reward model. In RQ4 the residual changes the Top-1 choice relative to the anchor in a small number of cases (as seen inTable 3). The learned update states are also distributed quite differently from the RQ3 reference states: PULSE predicts long-term revision for about half of updates (in App F), while the reference set labels few cases that way (in App G). An anchor-only row in Table 1, plus a predicted-vs-reference update-state comparison on the same instances, would make the paper's central claims much more convincing.

Questions for authors

How does the frozen anchor reward alone perform in each Table 1 setting? And can you show, perhaps with a few examples or an analysis by update state, how the predicted local vs. long-term states change what the criterion inducer and residual reward do downstream?

1 comment

Q1: It is not yet clear how much of the improvement comes from the two proposed mechanisms rather than from the trained anchor reward model. In RQ4 the residual changes the Top-1 choice relative to the anchor in a small number of cases (as seen inTable 3). The learned update states are also distributed quite differently from the RQ3 reference states: PULSE predicts long-term revision for about half of updates (in App F), while the reference set labels few cases that way (in App G). An anchor-only row in Table 1, plus a predicted-vs-reference update-state comparison on the same instances, would make the paper's central claims much more convincing.

A1: We thank the reviewer for these constructive suggestions. We have conducted the requested anchor-only evaluation and provide a matched-instance comparison of predicted and reference update states.

(1) Contribution beyond the anchor reward. We additionally evaluate Anchor-only, which uses the same frozen, profile-free anchor without the personalized residual, under all four settings in Table 1.

Backbone / SettingAnchor-only (Top-1 / MRR)PULSE (Top-1 / MRR)Δ\DeltaTop-1
Qwen3-4B / Seen90.12% / 0.945990.52% / 0.9466+0.40 pp
Qwen3-4B / Unseen88.14% / 0.932389.95% / 0.9436+1.81 pp
Llama-3.2-3B / Seen90.52% / 0.945992.68% / 0.9554+2.16 pp
Llama-3.2-3B / Unseen88.92% / 0.936090.83% / 0.9436+1.91 pp

PULSE consistently improves both Top-1 accuracy and MRR over Anchor-only, with Top-1 gains of 0.40–2.16 percentage points. These full-test results demonstrate that the improvements cannot be attributed to the trained anchor alone. Together with the module-level ablations in RQ2, they provide additional evidence for the contributions of the proposed mechanisms.

(2) Matched-instance update-state comparison. Appendix F reports PULSE's predictions over the full test set, whereas Appendix G constructs reference labels for a separate 175-instance diagnostic subset. Their state distributions therefore should not be directly compared. We now provide the following comparison using exactly the same 175 instances:

Update StatePULSE PredictionsLLM-adjudicated ReferenceHuman Reference
No Update455266
Local Refinement5111773
Long-term Revision79636

Notably, human validation identifies 36 Long-term Revision cases, compared with only six under the original LLM-adjudicated references. Against the human references, PULSE achieves 57.61% Macro-F1, compared with 54.30% under the original references. Furthermore, all 45 predicted No Update cases agree with both reference sets.

Importantly, PULSE's updater is trained using downstream preference-ranking utility, rather than direct supervision from update-state labels. Thus, the state-level analysis complements the downstream personalization evaluation, which directly measures the effectiveness of the resulting profile updates.

Reviewer_Qm3TOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PULSE, a framework for long-term LLM personalization that seperates when to update a user profile from when stored preferences should influence a response. Concretely, a profiles updater selects supporting interactions, predicts whether they warrant on update, a local refinement, or a long-term revision, and generates a profile statement linked to their evidence. A second component converts relevant profile information into request-specific evaluation criteria and learns a personalized residual reward that adjusts a frozen, profile-free anchor score. On HorizonBench, using Qwen3-4B and Llama-3.2-3B under seen- and unseen-user settings, PULSE reports the best overall response-ranking accuracy and MRR.

Critical strengths
  • The problem formulation is clear and practical. The paper distinguishes temporary requests from persistent preferences and separates accurate memory storage from appropriate memory use, which are two important personalization failure modes.
  • The evaluation is well structured. Experiments cover two model families, static/evolving preferences, and seen/unseen users. Profiles evolve using the model’s own predictions rather than reference-profile teacher forcing.
  • Beyond ranking accuracy, the paper evaluates update-state prediction, evidence support, criterion abstention, and beneficial versus harmful ranking changes.
Critical weaknesses
  • The evaluationn uses only 175 instances from 18 users, includinng six long-term-revision examples. References are entirely LLM-generated, without human validation, and update-state macro-F1 is only 54.3.
  • The attribution reward checks citation structure rather than semantic support, and an unconstrained residual can override the anchor’s quality preferences. The residual analysis shows only four corrected rankings and one newly introduced error, providing limited evidence for broad quality-preservation claims.
  • Ablations remove multiple design choices together, making it difficult to isolate evidence selection, update scope, criterion filtering, and residual scoring. Close alternatives such as PGMem and P-CHECK are discussed but not directly compared.
Questions for authors

see weaknesses above

2 comments

Q1: The evaluationn uses only 175 instances from 18 users, includinng six long-term-revision examples. References are entirely LLM-generated, without human validation, and update-state macro-F1 is only 54.3.

A1:We thank the reviewer for these valuable concerns. We would like to clarify the following three points.

(1) Evaluation scale. The 175 instances from 18 users are used specifically for the fine-grained evaluations in RQ3 and RQ4, rather than the entire evaluation. Our main experiments (Table 1) include 496 seen-user and 388 unseen-user test instances, evaluated across two backbone models and both static and evolving preferences. Thus, the overall personalization results are not based solely on the 175-instance subset.

(2) Human validation of reference labels. The original references were constructed using two independent GPT-5.5 annotation runs followed by adjudication. To address the reviewer's concern, we additionally conducted human validation on the same 175 instances. The results are as follows:

ReferenceNo UpdateLocal RefinementLong-term RevisionPULSE Macro-F1
LLM-adjudicated52117654.30%
Human Validation66733657.61%

Under human reference labels, 36 instances are classified as Long-term Revision, compared with six under the original LLM-adjudicated references. Moreover, the same PULSE predictions achieve 57.61% Macro-F1 against human references, providing additional human-annotated evaluation evidence beyond the original LLM-generated labels.

(3) Update-state prediction performance. Although the reported Macro-F1 is 54.3%, PULSE outperforms the Prompted Tri-State baseline (40.8%) by 13.5 percentage points under identical evaluation conditions. Furthermore, all 45 instances predicted as No Update by PULSE agree with both the LLM-adjudicated and human references, achieving 100% precision for No Update on this subset. These results provide evidence that the learned updater improves update-state prediction over direct prompting.

Q2: The attribution reward checks citation structure rather than semantic support, and an unconstrained residual can override the anchor’s quality preferences. The residual analysis shows only four corrected rankings and one newly introduced error, providing limited evidence for broad quality-preservation claims.

A2: We thank the reviewer for these insightful concerns. We clarify the roles of the attribution reward and personalized residual, and provide additional evidence regarding their effectiveness.

(1) Semantic support beyond citation structure. The attribution term (A(Z)A(Z)) measures structural provenance consistency, but it is not the sole signal used to optimize profile updates. As defined in Eq. (4), the updater's utility additionally incorporates current and future response-preference gains and preference-drift consistency. More importantly, we explicitly evaluate semantic evidence support using ESR, which assesses whether revised profile statements are supported by their cited interactions. In Table 2, PULSE achieves 82.2% ESR, compared with 72.4% for Prompted Tri-State, demonstrating improved semantic grounding beyond citation-format correctness.

(2) Why the residual can change anchor rankings. The residual is intended to correct cases where a response is generally appropriate but fails to satisfy an applicable user preference. Preventing it from changing the anchor's ranking would eliminate this corrective capability. Instead, PULSE keeps the profile-free anchor frozen and introduces the residual through request-specific personalized criteria. When no applicable criterion is generated, the residual is omitted, and the final ranking is determined entirely by the anchor.

(3) Additional evidence for selective correction. Table 3 provides a focused analysis of residual interventions: when personalization is warranted, the residual corrects four anchor errors without introducing harmful flips. To further assess its effectiveness beyond this diagnostic subset, we additionally evaluate the anchor-only baseline on the complete test sets, using the same evaluation protocol:

Backbone / SettingAnchor-only (Top-1 / MRR)PULSE (Top-1 / MRR)Δ\DeltaTop-1
Qwen3-4B / Seen90.12% / 0.945990.52% / 0.9466+0.40 pp
Qwen3-4B / Unseen88.14% / 0.932389.95% / 0.9436+1.81 pp
Llama-3.2-3B / Seen90.52% / 0.945992.68% / 0.9554+2.16 pp
Llama-3.2-3B / Unseen88.92% / 0.936090.83% / 0.9436+1.91 pp

PULSE consistently improves both Top-1 accuracy and MRR over the anchor-only baseline across all four settings. These full-test results complement the localized analysis in Table 3, providing broader evidence that the criterion-guided residual yields beneficial personalized corrections while retaining a fixed, profile-independent quality reference.

Q3: Ablations remove multiple design choices together, making it difficult to isolate evidence selection, update scope, criterion filtering, and residual scoring. Close alternatives such as PGMem and P-CHECK are discussed but not directly compared.

A3: We thank the reviewer for these suggestions. We clarify the objectives of our ablation design and the methodological differences from the suggested alternatives.

(1) Ablation design. RQ2 is specifically designed to evaluate the contributions of our two core mechanisms: Factorized Tri-State Profile Update and Criterion-Guided Personalized Residual Reward, rather than attributing performance changes to every internal design choice separately. Accordingly, we construct three controlled variants: w/o Factorized Update, w/o Criterion-Guided Reward, and w/o Both. This design directly evaluates whether each proposed mechanism, and their combination, contributes to the final ranking performance.

To further isolate the contribution of personalized residual scoring, we additionally conduct an anchor-only ablation, which retains the same frozen anchor but removes the criterion-guided residual. Compared with Anchor-only, PULSE improves Top-1 accuracy by 0.40/1.81 percentage points on Qwen3-4B (seen/unseen) and 2.16/1.91 percentage points on Llama-3.2-3B (seen/unseen), with consistent MRR improvements. This provides additional evidence of the residual's contribution beyond the original module-level ablations.

(2) Comparisons with PGMem and P-CHECK. Although closely related, these methods differ in their core functionality and required information.

  • PGMem focuses on evidence-grounded persona memory construction and validity-aware retrieval through a heterogeneous persona–memory graph. Although it tracks evolving persona states, its primary contribution lies in memory organization and retrieval rather than modeling how stored user preferences should influence personalized response evaluation.

  • P-CHECK assumes user histories containing previously labeled chosen–rejected preference pairs and additionally constructs cross-user contrastive responses for criterion-weight supervision. In contrast, PULSE learns profile updates from chronological user–assistant interactions and implicit feedback, without requiring pairwise preference annotations in the user's interaction history. Applying the original P-CHECK formulation would therefore require additional historical preference-pair information beyond PULSE's input.

Comments on the paper

1 comment

We sincerely thank all reviewers for their thorough evaluations and constructive feedback. We appreciate their recognition of the importance of distinguishing reliable profile updates from selective preference utilization, as well as the interpretability of our provenance-aware framework and its consistent performance across different models and user settings. To further address the reviewers' concerns and strengthen the empirical evidence for PULSE, we conducted additional evaluations and provide the following results and clarifications.

  1. Human Validation of Profile-Update States (Reviews Qm3T and zeze): We conducted human validation on the 175 evaluation instances. The human references identify 36 Long-term Revision cases, compared with six under the original LLM-adjudicated references. PULSE achieves 57.61% Macro-F1 against the human references, compared with 54.30% against the original references.
  2. Anchor-only Comparisons (Reviews Qm3T and FKFF): We evaluated the frozen anchor reward independently across four backbone and user settings. PULSE consistently outperforms Anchor-only in both Top-1 accuracy and MRR, with Top-1 improvements of 0.40–2.16 percentage points.
  3. Query-only Criterion Baseline (Review zeze): We introduced a controlled baseline that constructs evaluation criteria solely from the current request, without historical preferences. PULSE achieves consistent Top-1 improvements of 0.37–1.95 percentage points, demonstrating the additional value of historical personalization.
  4. Matched Update-State Analysis (Review FKFF): We provide a comparison of PULSE's predicted update states against both LLM-adjudicated and human-validated references on the same 175 instances, clarifying the state-distribution differences discussed in the review.

Detailed explanations and experimental results are provided in our responses to the corresponding reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.