acceptodds
Under review as a conference paper at ICLR 2027

PersonaTrust: Benchmarking Trustworthy Personalized MLLM Assistants

Abstract

As MLLMs become lifelong personalized companions, the reliability of their customized responses assumes unprecedented importance: users expect them not only to provide highly tailored assistance, but also to honestly acknowledge when user-specific evidence and information are insufficient, thereby avoiding misleading or unsafe recommendations. To bridge this gap, we propose PersonaTrust, the first benchmark dedicated to trustworthiness throughout MLLM personalization. In contrast to existing MLLM trustworthiness benchmarks that mainly assess model reliability on questions with well-defined answers, PersonaTrust focuses on personalized questions with inherently open-ended answer spaces. Therefore, trustworthiness is evaluated by whether the model can faithfully narrow this space according to the available personalized evidence. We observe that existing frontier MLLMs and personalization methods fall short on PersonaTrust, particularly by creating a false sense of familiarity with users, e.g., providing generic recommendations without clarifying that they are not grounded in sufficient personalized evidence. To remedy this, we propose a simple plug-and-play method, PR-Calibrator, which equips these assistants with two-layer calibration safeguards before and after the original pipeline, improving reliability while largely preserving their personalization capabilities. Experiments on PersonaTrust demonstrate that PR-Calibrator clearly improves the trustworthiness of existing personalization methods, while also highlighting the need for future MLLM assistants to better recognize the boundaries of their user-specific knowledge.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Long-Context MLLM Personalization, Personalization Trustworthiness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y7UTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PersonaTrust, a benchmark for evaluating trustworthiness of personalized MLLMs. The dataset categorizes user queries into four types (PIQ, EPQ, GPQ, NQ) and tests models under varying levels of evidence availability. Findings show that frontier models frequently fail to abstain or reveal missing context, often hallucinating personalized details instead. To address this, the authors propose PR-Calibrator, a 2-stage wrapper involving query diagnosis and calibration, which significantly improves trustworthiness on complex queries while preserving the model's baseline personalization abilities.

Critical strengths
  1. The paper identifies a gap that existing MLLM personalization benchmarks have overlooked: whether personalized assistants remain reliable when user-specific evidence is insufficient or conflicting.

  2. The benchmark is well designed, decomposing trustworthy personalization into four question types (PIQ/EPQ/GPQ/NQ) that capture distinct evidence-dependency scenarios.

Critical weaknesses
  1. The current evaluation does not sufficiently decouple a model's general personalization capabilities from the specific trustworthiness issues measured by PersonaTrust. Because the benchmark is derived from Persona-MME and LCMP, the authors should report the models' baseline performance on these source datasets. This comparison is necessary to clarify whether the observed failures stem from a specific vulnerability to overstated familiarity, or simply from fundamentally weak personalization skills.

  2. The proposed PR-Calibrator introduces a non-marginal computational overhead, which raises practical concerns for real-world serving. Furthermore, the evaluation lacks comparison against more straightforward, lightweight baselines. For instance, the authors should test whether a strong system prompt (e.g., explicitly instructing the model to abstain or provide a disclosed general fallback when context is missing) can mitigate the issue in a single step. Additionally, a simpler two-stage approach can be evaluated, where generation is directly conditioned on the upfront diagnosis step. The necessity of generating an initial draft only to correct it retrospectively in a third stage is not adequately justified.

Questions for authors
  1. In Figure 3, GFP is fairly high while FT is low, meaning the models do provide a general response in most cases. Why, then, are P-ACC and A-ACC on GPQ so much lower than GFP? Does a response count as incorrect whenever it fails to explicitly disclose that it is a general fallback, even if the fallback content itself is reasonable?

  2. For GPQ, what do the authors think about separately checking whether the model gives a general fallback response versus a (fabricated) personalized response? This would help distinguish “answered generically without disclosure” from “pretended to be personalized.”

  3. What is the accuracy of the UPD step itself? How well does it categorize incoming queries into the question types at this stage?

  4. The ablation study lacks a description of how the process actually runs when a part of the component is present. For example, in the “+ UPD Step” row, does the pipeline execute only steps 1 and 2 in Figure 5 and stop? If so, where is the diagnosis information injected—into the prompt of the generation step, or elsewhere?

  5. Regarding naming: “Never-answerable Questions” does not convey what the category actually tests. A name that reflects the conflict between the query’s premise and the past context might be more intuitive.

5 comments
AuthorsOpenReview ↗

Response to Reviewer y7UT (1/3)

(Part 1/3)

Thank you for the careful reading and for recognizing that PersonaTrust targets an overlooked gap, whether personalized assistants stay reliable when user evidence is insufficient or conflicting, with a well-designed set of four question types (PIQ/EPQ/GPQ/NQ). We also appreciate the concrete suggestions on baselines, metrics, and naming.

[W1] The evaluation does not decouple general personalization ability from the trustworthiness issues. Please report the models' performance on the source datasets (Persona-MME, LCMP).

We agree that distinguishing general personalization ability from trustworthiness is important. However, we believe there may be a misunderstanding, as our PersonaTrust benchmark is already designed to disentangle these two aspects.

  1. The Full, Partial, and None settings separate the two aspects. Specifically, for each EPQ and GPQ question, we construct a Full setting, in which all the required evidence remains in the interaction history, and Partial and None settings, in which we mask some or all of that evidence. Because the model and the question stay the same across the three settings, accuracy under Full (F-ACC) reflects the model's ability to use sufficient personalized evidence, whereas accuracy under Partial and None (P-ACC and A-ACC) reflects whether it recognizes that evidence is missing (Section 3.2).
  2. Stronger personalization ability does not imply trustworthiness. To verify this, we compare F-ACC and P-ACC for each model in Table 1. If trustworthiness simply followed personalization ability, the models with the highest F-ACC would also achieve the highest P-ACC. However, this is not the case. For example, PersonaVLM achieves the highest U-Bench EPQ F-ACC (18.67), yet its P-ACC is only 2.02.
  3. Models fail even where no personalization ability is needed. Moreover, we examine two behaviors that do not depend on using personal evidence. First, in the EPQ None setting, all the required evidence is removed, so the model does not need to retrieve anything and only has to abstain without fabricating user facts. Even in this simple case, all models perform poorly on U-Bench EPQ A-ACC (Table 1). Second, on GPQ, we separately measure whether a model provides a general answer (GFP) and whether it discloses that the answer is not personalized (FT). As shown in Figure 3, models often provide the general answer but rarely disclose it. Since neither abstaining nor disclosing depends on using personal evidence, these failures cannot be attributed to weak personalization ability.

To complement this analysis, we report performance on the source benchmarks in the table below. To relate these numbers to trustworthiness, the last column first takes the EPQ and GPQ questions that a model answers correctly under Full, and then reports how often the model fails on these same questions under Partial and None.

ModelPersona-MME acc.LCMP ACC-CPartial/None fail rate when Full passes
PersonaVLM71.27——
TAME—70.76—
Qwen3.8-27B77.5188.89 (hard)87.7 (206/235, U-Bench) / 95.0 (134/141, C-Bench)

For PersonaVLM and TAME, source-benchmark results come from their papers, averaged over Persona-MME 32K/128K and LCMP-E/LCMP-H. For Qwen3.8-27B, we evaluate both source benchmarks ourselves.

Thus, Qwen3.8-27B is strong on both source benchmarks, yet on the very questions it answers correctly under Full, 87.7% (U-Bench) and 95.0% (C-Bench) of its Partial and None responses still fail. We will add these results for all models in the revision.

AuthorsOpenReview ↗

Response to Reviewer y7UT (2/3)

(Part 2/3)

[W2] (1) PR-Calibrator introduces a non-marginal computational overhead, which raises practical concerns for real-world serving. (2) The evaluation lacks a strong-system-prompt baseline, e.g., one that explicitly instructs the model to abstain or give a disclosed general fallback when context is missing. (3) A simpler two-stage approach, where generation is directly conditioned on the upfront diagnosis, should also be evaluated. (4) The need to generate an initial draft and then correct it in a third stage is not adequately justified.

  1. We appreciate this practical concern and acknowledge that PR-Calibrator adds overhead, which is a trade-off for training-free trustworthy personalization. Specifically, diagnosis is independent of the draft to avoid being influenced by an unreliable answer. Meanwhile, preserving the original generation retains the base assistant's personalization pipeline, while PC verifies the draft against the history. Moreover, UPD reads only the query and can run in parallel with generation, with PC following both.

  2. We compare a single-call strong-system-prompt baseline with Qwen3.8-27B and PR-Calibrator. The prompt requires grounding user facts in the history, acknowledging missing facts, disclosing general advice, and checking unsupported premises.

    The neutral-prompt and PR-Calibrator rows use the reported Table 2 results, and Avg is the macro-average of its eight metrics.

    Qwen3.8-27BU-Bench EPQ A-ACCU-Bench GPQ A-ACCU-Bench AvgC-Bench Avg
    Neutral prompt23.0710.3125.6731.52
    Strong system prompt46.4740.8341.5335.60
    PR-Calibrator64.0766.1650.6937.69
  3. Thank you for this suggestion, and we would like to clarify that this simpler two-stage approach has already been evaluated in Table 3 (the "+ UPD Step" row, see Q4), where the complete PR-Calibrator achieves better overall trustworthiness performance.

  4. We agree that this design choice needs a clearer justification. Specifically, we draft first because the two-stage variant improves trustworthiness but halves GPQ F-ACC (28.44→14.4628.44 \to 14.46), since with the rules in the generation prompt the model turns conservative even when the evidence is present. In contrast, drafting first and calibrating afterwards keeps the original pipeline unchanged (GPQ F-ACC 29.69) and edits only what the history does not support. At the same time, revision without the diagnosis is not enough either, as GPQ A-ACC reaches 15.16, versus 66.16 with it.

[Q1] In Figure 3, GFP is high and FT is low. Why are GPQ P-ACC and A-ACC so much lower than GFP? Is a response counted as incorrect whenever it does not disclose that it is a general fallback, even if the fallback itself is reasonable?

Thank you for this careful question, and your understanding is correct. A reasonable general fallback is still counted as incorrect if the response does not disclose that it is not personalized.

Specifically, GPQ A-ACC counts a response as correct only when it meets all four conditions (Appendix B.2). It does not abstain, it gives a general answer (GFP), it states that the answer is not personalized (FT), and it does not present generic content as personalized. Since both GFP and FT are required, A-ACC cannot exceed the lower of the two, and therefore the low FT caps A-ACC even though GFP is high. GPQ P-ACC additionally requires the response to use at least one available evidence atom and to make no fabricated persona claim.

We count undisclosed fallbacks as failures because the user asked for advice based on their own history. Hence an undisclosed generic answer is read as tailored advice, which is the false sense of familiarity that PersonaTrust targets (Section 3.3). Nevertheless, the fallback itself is still credited by GFP, which is why Figure 3 reports GFP and FT separately.

AuthorsOpenReview ↗

Response to Reviewer y7UT (3/3)

(Part 3/3)

[Q2] Could GPQ separately check "answered generically without disclosure" versus "pretended to be personalized"?

Thank you, this is a useful distinction, and the judge already records it. Specifically, for GPQ, it returns "gave general answer", "disclosed not personalized", and "fake personalization" separately (Table 5). Based on these records, we split the U-Bench GPQ responses under the None setting into the two cases, where each response is counted once.

U-Bench GPQ (None)Answered generically without disclosurePretended to be personalized
GPT-5.6-Sol46.426.3
Qwen3.8-27B13.551.6

Thus, the two models fail in different ways. GPT-5.6-Sol mostly answers generically without disclosure, while Qwen3.8-27B mostly presents generic content as personalized. We will add this split for all models in the revision.

[Q3] What is the accuracy of the UPD step itself?

Thank you for the question, which we answer by measuring UPD's classification accuracy on all 1,189 U-Bench and all 493 C-Bench queries with Qwen3.8-27B. Note that UPD sees only the query, so EPQ and NQ cannot be told apart at this stage (Section 4.2), and thus an NQ routed to EPQ counts as correct, because the EPQ calibration request also carries the NQ handbook (Appendix G).

SubsetPIQEPQGPQNQ (→\to EPQ)All
U-Bench94.795.772.795.089.7
C-Bench74.493.592.692.085.7

Thus, UPD diagnoses most queries correctly, with an overall accuracy of 89.7% on U-Bench and 85.7% on C-Bench. The remaining errors concentrate in two cases. On U-Bench, they mostly route recall-style GPQs to EPQ (GPQ accuracy 72.7%). On C-Bench, they mostly route advice-style PIQs to GPQ (PIQ accuracy 74.4%), since query-only diagnosis can mistake generic advice for personal advice.

[Q4] The ablation study does not describe how the pipeline runs when only some components are present. For example, does the "+ UPD Step" row execute only Steps 1 and 2 of Figure 5 and stop, and if so, where is the diagnosis injected?

We apologize for the missing description, and your reading of the "+ UPD Step" row is correct. Specifically, this variant runs Steps 1–2, where the diagnosis (type, needs, premises), the corresponding handbook, and grounding rules G1–G4 are injected into the generation prompt, producing the final response without post-generation calibration. In contrast, the "+ PC Step" variant runs Steps 2–3, where the calibration step receives all handbooks without an upfront diagnosis and determines the appropriate case itself. Thus, both variants require only two model calls. We have clarified how both variants run in Section 5.3 (revised PDF).

[Q5] "Never-answerable Questions" does not convey what the category tests, and a name reflecting the conflict between the premise and the past context may be more intuitive.

Thank you for the suggestion on the name "never-answerable", which we chose to contrast NQ with EPQ and GPQ. Specifically, for those two types, the right response depends on how much evidence is available, while an NQ cannot be answered as posed even with the complete history and has no general fallback (Section 3.1), because its premise conflicts with, or is not supported by, the history. That said, we agree that the name describes the outcome rather than the cause. Therefore, if you find it clearer, we are happy to rename the category in the revision, e.g., to Persona-Conflict Questions (PCQ), which matches the existing PCA-ACC metric.

Reviewer_3VLmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Personalized MLLM assistants answer from a user’s long-term multimodal interaction history. When the history lacks the needed information, or the user’s question contradicts it, these assistants tend to answer as if they knew the user. They create a “false sense of familiarity” and may mislead users. The paper argues that existing trustworthiness and abstention studies focus on questions with a single well-defined answer, and do not cover open-ended personalized questions.

They built PersonaTrust, on top of Persona-MME and LCMP. It has four question types: PIQ (answerable without persona evidence), EPQ (the model should abstain if evidence is missing), GPQ (a general fallback is acceptable if disclosed as non-personalized) and NQ (the question contains a premise that conflicts with the history). Each EPQ and GPQ question is linked to its evidence in the history, which lets them mask that evidence to create Full, Partial and None settings. An MLLM judge scores the open-ended answers. They propose PR-Calibrator, a prompt-based wrapper around an existing assistant. It first diagnoses what kind of question the query is (UPD step), lets the assistant respond as usual, and then checks the response against the history and rewrites it using handbooks of expected behavior (PC step).

They find Frontier MLLMs and two personalization methods (PersonaVLM and TAME) behave poorly when evidence is partial or missing, or when a premise conflicts with the history. They often give a general answer but rarely disclose that it is not personalized. They also find PR-Calibrator substantially improves these behaviors, while leaving accuracy on questions with full evidence and on PIQ largely unchanged. The authors conclude that future assistants need to better recognize the limits of their knowledge about a user.

Critical strengths

Each EPQ and GPQ question in PersonaTrust is tied to the specific evidence atoms and history segments it needs. Masking them lets the same question be asked with all, some or none of its evidence (the Full, Partial and None settings). This is more informative than a single answerable/unanswerable split. Appendix B.1 also describes audits for leaked or restated evidence and a sufficiency check on every Partial instance. These make the evaluation controllable, and I expect other researchers to reuse it.

The GPQ construct, together with the GFP and FT metrics, targets the “false sense of familiarity” the paper is about. It separates giving a general answer (GFP, high for the evaluated systems) from saying the answer is not personalized (FT, low). The same failure appears across frontier MLLMs, PersonaVLM and TAME, which suggests a systematic gap and not an artifact of one model or prompt.

Critical weaknesses

Some of the results are hard to interpret. With the complete history, the best model reaches only 17.47 F-ACC on U-Bench EPQ (Qwen3.8-27B: 12.67). That seems to be too low for the Full setting to serve as the reference for “inherent answerability” that Section 3.2 describes. Low P-ACC and A-ACC cannot be separated into unsupported persona claims, failures to retrieve the evidence, or strict key-point scoring. The claim that F-ACC and PIQ ACC stay “largely unchanged” is weak when F-ACC starts so low, and some drops are visible in Table 2 (GPT-5.6-Sol PIQ ACC on C-Bench 96.76→91.03; TAME GPQ F-ACC 24.10→19.74). Each cell is a single average over 5 runs, with no standard deviation or confidence interval, and several C-Bench slices are small (68 EPQ and 78 GPQ questions), so differences of a few points could be noise. Table 2 also gives Qwen3.8-27B and PersonaVLM, each with PR-Calibrator, identical EPQ P-ACC (14.11) and NQ PCA-ACC (92.87) on U-Bench.

The evaluation also has gaps. The judge’s reliability (Appendix B.2) is shown on other models’ responses and not against human annotators, and the exact-substring quote rule for supported claims may count correct paraphrases as unsupported. NQ is defined as a premise conflicting with the history, but some C-Bench NQs state premises “never established” (Appendix B.1). Since LCMP attributes are “frequently updated” by design, a differing statement may be a valid update, and there is no control for this.

The evidence that PR-Calibrator’s three-step chain is needed is thin. The base assistants get a neutral instruction (Figure 11) and are not compared with simpler alternatives at equal cost, such as giving them an abstention and disclosure instruction. The handbooks restate the benchmark’s own expected behaviors. The gain varies widely by base assistant (GPQ A-ACC on U-Bench: Qwen3.8-27B 10.31→66.16, GPT-5.6-Sol 9.97→13.01) without explanation. The ablation (Table 3) covers only Qwen3.8-27B on U-Bench, and UPD’s classification accuracy is not reported. The PC step reads “the available history”, bypassing the memory pipelines of PersonaVLM and TAME, and the 2.0× token cost is reported only for Qwen3.8-27B. The “first benchmark” claim also does not cite related work such as LongMemEval and MirageBench.

Questions for authors

The tables report a single number per cell, with no variance or confidence intervals. The authors average 5 runs but never report the spread across them. Several slices are small (C-Bench has 68 EPQ and 78 GPQ questions), so differences of a few points could be noise. This matters for the claim that Full-evidence accuracy is “largely preserved”, since the changes there are only a few points. Generally, it will be helpful to see evidence added to justify the reliability of results.

The best model gets only 12–18% F-ACC on EPQ even with complete evidence. What do the Full-setting failures consist of (retrieval misses, strict key-point scoring, fabrication)? Given this, what does Full establish about answerability, and how should Partial and None results be read?
 The base models receive a neutral prompt with no abstention hint. How do the results compare with simpler or existing alternatives at equal cost? Wondering if the three-step chain (diagnosis, personalized generation, and post-generation calibration) is necessary.

The same method gives GPQ-None 10.3→66.2 on Qwen but 10.0→13.0 (much less) on GPT-5.6-Sol. Wonder what explains this, and does the ablation hold for other backbones and for C-Bench?

A minor check: Table 2 has identical values for Qwen3.8-27B and PersonaVLM with PR-Calibrator on U-Bench EPQ-Partial (14.11) and NQ (92.87). Are these coincidences?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3VLm (1/4)

(Part 1/4)

Thank you for the detailed and constructive review, and for recognizing that the evidence-linked Full/Partial/None settings make the evaluation controllable and reusable, and that GPQ with GFP and FT exposes the false sense of familiarity as a systematic gap across models. We also appreciate how closely you checked our tables.

[W1/Q2] With the complete history, the best model reaches only 17.47 F-ACC on U-Bench EPQ, which seems too low for Full to serve as the reference for "inherent answerability" (Section 3.2). (1) What do the Full failures consist of (retrieval misses, strict key-point scoring, or fabrication)? (2) Low P-ACC and A-ACC cannot be separated into unsupported persona claims, failures to retrieve the evidence, or strict key-point scoring. (3) Given the low F-ACC, how should the Partial and None results be read?

  1. Full remains a valid reference, since answerability is ensured by construction and not by F-ACC. Specifically, we audit every EPQ and GPQ during construction to confirm that the full history contains all the evidence needed to answer it (Appendix B.1). Therefore, the low F-ACC reflects how hard U-Bench is for current models, as its EPQs combine evidence from at least two sessions in histories of 178 turns on average (Table 4). In comparison, on C-Bench with 40-turn histories, the same MLLMs reach 42.35–51.47 EPQ F-ACC.

    To show what the Full failures consist of, we take the failed Full responses on U-Bench and check which pass condition each one violates (Appendix B.2). Since one response can violate several conditions, the first three columns can sum to more than 100%. The last column is the average share of key points covered over all Full responses.

    ModelTypeMissing key pointFabricated claimAbstainedKey-point recall (all Full)
    GPT-5.6-SolEPQ90.4%52.2%10.0%43.1%
    GPQ80.1%41.1%0.0%72.4%
    Qwen3.8-27BEPQ89.1%52.1%19.5%41.6%
    GPQ62.0%77.1%0.0%70.4%

    As shown, the Full failures are mainly retrieval misses, often combined with fabrication. In contrast, strict key-point scoring is not the main cause, because the models cover fewer than half of the EPQ key points overall, rather than narrowly missing one.

  2. We agree that these causes should be separated, and we first note that key-point scoring does not enter P-ACC or A-ACC, since P-ACC needs only one available atom, and A-ACC needs none (Appendix B.2). Furthermore, we decompose the failed Partial and None responses on U-Bench into three causes.

    • Retrieval only. The response uses none of the remaining evidence but meets every other condition.
    • Missing evidence not acknowledged. The response does not abstain, point out the missing piece, or disclose that its answer is general.
    • Unsupported persona claim. The response states a user fact that the history does not support.

    Each cell is the percentage of failed responses with that cause, given as the range over GPT-5.6-Sol and Qwen3.8-27B. Since one response can have several causes, a column can sum to more than 100%.

    Failure causeEPQ PEPQ NGPQ PGPQ N
    Retrieval only10–18%n/a1–2%n/a
    Missing evidence not acknowledged82–88%92%95–96%86–99%
    Unsupported persona claim52–58%67–75%33–62%32–59%

    As shown, retrieval alone explains a minority of P-ACC failures and none of A-ACC. Instead, most failures involve unacknowledged missing evidence or unsupported persona claims. We have added both decompositions to Appendix B.2 (revised PDF) and will extend them to all models on both subsets.

  3. The Partial and None results can be read independently of the low F-ACC. As noted in (2), they do not depend on key-point scoring. Instead, they test whether a model answers the supported part and then abstains (EPQ) or gives disclosed general advice (GPQ) for the rest, without inventing user facts. Moreover, a higher F-ACC does not lead to better results here. For example, PersonaVLM has the highest U-Bench EPQ F-ACC (18.67), yet only 2.02 P-ACC (Table 1), which shows that stronger Full performance does not ensure trustworthiness with missing evidence.

AuthorsOpenReview ↗

Response to Reviewer 3VLm (2/4)

(Part 2/4)

[W1/Q1] The claim that F-ACC and PIQ ACC stay "largely unchanged" is weak, and some drops are visible (GPT-5.6-Sol C-Bench PIQ 96.76→91.0396.76 \to 91.03 and TAME GPQ F-ACC 24.10→19.7424.10 \to 19.74). Each cell averages 5 runs with no std or CI, and some C-Bench slices are small (68 EPQ, 78 GPQ), so a few points could be noise.

We agree that the phrase "largely unchanged" requires more careful wording, as some drops in PIQ ACC and F-ACC are indeed visible. What we intended to convey is that these drops are much smaller than the gains in trustworthiness. For example, on C-Bench, GPT-5.6-Sol's PIQ ACC decreases by 5.73 points (96.76→91.0396.76 \to 91.03), while its trustworthiness metrics improve by 11.61–15.76 points. Similarly, TAME's GPQ F-ACC decreases by 4.36 points (24.10→19.7424.10 \to 19.74), whereas its GPQ P-ACC and A-ACC improve by 10.99 and 22.30 points. We have revised the wording accordingly in Section 5.2 (revised PDF).

Moreover, we agree that differences of a few points can be noise on the small C-Bench slices. Therefore, we report every Qwen3.8-27B cell in Table 2 below as mean ± sample standard deviation across five runs.

MethodPIQEPQ FEPQ PEPQ NGPQ FGPQ PGPQ NNQ
Qwen3.8-27B, U-Bench96.80±0.5112.67±0.532.42±0.6423.07±0.9228.44±0.962.07±0.6510.31±0.9029.60±0.64
+ PR-Calibrator96.93±0.4910.87±0.6914.11±0.9064.07±1.3029.69±0.9030.83±1.0866.16±1.0892.87±0.69
Qwen3.8-27B, C-Bench84.97±0.9746.18±2.231.36±0.638.82±2.3322.31±2.331.99±0.668.72±1.6777.78±0.98
+ PR-Calibrator88.97±1.1242.65±2.3314.55±1.2415.00±1.9219.74±1.468.08±0.8617.69±1.9094.81±0.70

As shown, every Partial, None, and NQ gain is more than twice the standard deviation, including on the small C-Bench slices.

[W1/Q4] Qwen3.8-27B and PersonaVLM with PR-Calibrator have identical U-Bench EPQ P-ACC (14.11) and NQ PCA-ACC (92.87). Are these coincidences?

Yes, we verified from the logs that this is a coincidence. Both runs pass 175 of the 1,240 scored EPQ-Partial instances (248×5248 \times 5 runs) and 1,393 of the 1,500 NQ instances (300×5300 \times 5). Their other cells differ (e.g., EPQ A-ACC 64.07 vs. 52.13).

[W2] (1) The judge's reliability (Appendix B.2) is shown on other models' responses, not against human annotators. (2) The exact-substring quote rule may count correct paraphrases as unsupported. (3) NQ is defined by a premise that conflicts with the history, but some C-Bench NQs state premises "never established" (Appendix B.1). Moreover, since LCMP attributes are "frequently updated" by design, a conflicting statement may be a valid update, and there is no control for this.

  1. We agree that the judge should also be validated against human annotators, and we will add a human study in which two annotators label a sample of responses from U-Bench and C-Bench. We will then report their agreement with the judge (Cohen's κ\kappa), including how often correct paraphrases are counted as unsupported (point 2).
  2. We apologize for the lack of clarity. The exact-substring rule applies only to the verifier's evidence quote, not to the evaluated response, which can freely paraphrase the history. Specifically, the verifier must quote the history exactly when it confirms that a flagged claim is supported, which prevents it from inventing evidence. Therefore, this rule does not count a correct paraphrase as unsupported.
  3. Thank you for these related observations. We agree that LCMP attributes are frequently updated by design, and this is the reason why some C-Bench NQs use never-established premises. Specifically, some C-Bench histories are short and contain too few stable attributes that can be safely changed into an unambiguous contradiction. For these cases, we therefore use premises that were never established in the history, so that a valid update is not treated as a contradiction. As a result, none of the C-Bench NQs involves a conflicting value that could be read as a legitimate user update, and the expected behavior remains to recognize that the premise is unsupported. We will clarify this distinction and, in future extensions, expand the source histories so that all NQs use explicit contradictory premises while further controlling for legitimate attribute updates.
AuthorsOpenReview ↗

Response to Reviewer 3VLm (3/4)

(Part 3/4)

[W3/Q2] (1) The base assistants get a neutral instruction (Figure 11) and are not compared with simpler alternatives at equal cost, e.g., an abstention and disclosure instruction. Is the three-step chain necessary? (2) The handbooks restate the benchmark's own expected behaviors.

  1. Yes, the three-step chain is necessary, and simpler alternatives fall short of it. To verify this, we first evaluate the instruction you describe, a single-call strong system prompt, which instructs the base assistant to acknowledge missing facts, disclose general advice, and check unsupported premises. We also evaluate Self-Refine with the same rules, which uses three calls as PR-Calibrator does. Each score below is the macro-average of the eight Table 2 metrics (Qwen3.8-27B).

    MethodCallsU-BenchC-Bench
    Neutral prompt (base)125.6731.52
    Strong system prompt141.5335.60
    Self-Refine343.0135.81
    PR-Calibrator350.6937.69

    As shown, the instruction alone helps, yet PR-Calibrator scores highest on both subsets, including at equal cost. Moreover, our ablation (Table 3) already covers two cheaper variants of the chain. Specifically, "+ UPD Step" places the diagnosis output, the corresponding handbook, and G1–G4 in the generation prompt, while "+ PC Step" performs a single post-generation revision with all handbooks and no diagnosis. Both need only two calls and use the same guidelines, yet the complete PR-Calibrator outperforms them across all Partial, None, and NQ settings, which shows that upfront diagnosis and post-generation calibration complement each other.

  2. We appreciate this observation, and this alignment is intentional. We believe that the four question types in PersonaTrust capture the key scenarios of trustworthy personalized responding from the perspective of personalized evidence sufficiency and consistency. Therefore, it is natural and reasonable for the handbooks to reflect the corresponding expected behaviors. Importantly, these guidelines represent general principles of trustworthy personalization, and our experimental results further demonstrate their effectiveness in improving the trustworthiness of existing personalized assistants.

[W3/Q3] (1) The gain varies widely by base assistant without explanation (on U-Bench GPQ A-ACC, Qwen3.8-27B 10.31→66.1610.31 \to 66.16, GPT-5.6-Sol 9.97→13.019.97 \to 13.01). (2) The ablation covers only Qwen3.8-27B on U-Bench. (3) UPD's classification accuracy is not reported.

  1. This gap mainly reflects a cautious diagnosis, while the calibration itself remains effective. To verify this, we trace how UPD diagnoses GPT-5.6-Sol's GPQ-None queries. Specifically, it assigns 78% of them to EPQ, the most conservative type (Appendix C.3), whose handbook asks the model to abstain when the evidence is missing. As a result, the calibrated response errs on the safe side and mostly abstains, which avoids the false sense of familiarity but is not credited by GPQ A-ACC, since this metric expects a disclosed general answer. In contrast, on the queries diagnosed as GPQ, the pass rate rises from 6.5% to 40.3%. Similarly, on NQ, where the EPQ diagnosis is the intended one, GPT-5.6-Sol improves from 39.53→73.7339.53 \to 73.73. For Qwen3.8-27B, UPD diagnoses most GPQ queries as GPQ (see (3)), and hence its gain on this cell is larger.

  2. We extend the ablation to GPT-5.6-Sol on U-Bench and Qwen3.8-27B on C-Bench.

    VariantPIQEPQ FEPQ PEPQ NGPQ FGPQ PGPQ NNQ
    GPT-5.6-Sol, U-Bench100.0017.477.9017.6747.3411.079.9739.53
    + UPD Step99.6716.339.1924.5341.8717.6911.9069.27
    + PC Step100.0017.609.8425.8045.1218.5112.3971.53
    + PR-Calibrator100.0017.8010.5628.0046.7120.0813.0173.73
    Qwen3.8-27B, C-Bench84.9746.181.368.8222.311.998.7277.78
    + UPD Step88.2238.5310.9113.5317.447.1515.6492.59
    + PC Step87.4643.2411.8212.0620.266.6213.8593.21
    + PR-Calibrator88.9742.6514.5515.0019.748.0817.6994.81

    In both settings, PR-Calibrator improves every Partial, None, and NQ cell over the base.

  3. We agree that this accuracy should be reported, and we measure it on all 1,189 U-Bench and 493 C-Bench queries (Qwen3.8-27B) below. Note that UPD sees only the query, so EPQ and NQ cannot be told apart (Section 4.2). Therefore, an NQ routed to EPQ counts as correct, because the EPQ calibration request also carries the NQ handbook (Appendix G).

    SubsetPIQEPQGPQNQ (→\to EPQ)All
    U-Bench94.795.772.795.089.7
    C-Bench74.493.592.692.085.7
AuthorsOpenReview ↗

Response to Reviewer 3VLm (4/4)

(Part 4/4)

[W3] (1) The PC step reads "the available history", bypassing the memory pipelines of PersonaVLM and TAME. (2) The 2.0×2.0\times token cost is reported only for Qwen3.8-27B. (3) The "first benchmark" claim does not cite LongMemEval and MirageBench.

  1. Thank you for raising this concern, and we clarify that bypassing the memory pipelines is an intentional design choice. Since PersonaVLM and TAME employ different and relatively complex memory-processing mechanisms, our plug-and-play method uses the raw interaction history as a unified evidence source for calibration. This design avoids coupling PR-Calibrator to specific memory architectures and helps disentangle the limitations of the original memory pipelines from the effectiveness of our calibration mechanism.

  2. We agree that the cost should not be reported for Qwen3.8-27B alone, and the table below gives the prompt-token cost of PR-Calibrator relative to each base assistant, from the recorded usage.

    Base assistantU-BenchC-Bench
    Qwen3.8-27B2.05×2.05\times2.10×2.10\times
    PersonaVLM1.11×1.11\times—
    TAME—1.9×1.9\times

    As shown, the cost stays around 2×2\times for Qwen3.8-27B on both subsets and for TAME, and most of it comes from the PC step, which reads the history again. For PersonaVLM, the ratio is lower because its own memory construction already takes most of the tokens.

  3. Thank you for these references, both of which we now cite and discuss in Related Work (revised PDF). LongMemEval evaluates the long-term memory of text chat assistants, and its abstention questions ask about information never mentioned, with a single expected behavior (abstain). MirageBench (arXiv) measures how personalized LLMs fabricate user attributes. Compared with them, PersonaTrust differs in three ways. Specifically, it offers (a) multimodal histories and queries, (b) open-ended questions whose trustworthy behavior depends on the type (answer without personalizing for PIQ, abstain for EPQ, a disclosed general fallback for GPQ, conflict detection for NQ), and (c) the same question evaluated under Full, Partial, and None by masking traceable evidence, together with a plug-and-play mitigation.

Reviewer_7j4yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how personalized multimodal assistants should answer when user information is missing or conflicting. It introduces PersonaTrust, a benchmark with 1,682 questions across 150 personas spread across 4-different question types (as defined in the paper). The questions were generated by an MLLM and manually checked by the authors.

It also proposes PRCalibrator, which adds prompting steps before and after answer generation to check what personal evidence is needed and available. Experiments show better handling of missing evidence and conflicting claims, though accuracy sometimes drops when full evidence is available.

Critical strengths
  1. The paper defines four clear question categories based on the personal evidence needed and the appropriate response. The categories are intuitive, but making them explicit gives researchers a useful framework for studying trustworthy personalization.

  2. The evaluation goes beyond overall answer accuracy. It tests several MLLMs with full, partial, and missing evidence, and measures whether they answer, abstain, or clearly offer general advice. This helps identify where each system fails and gives future work a concrete benchmark to build on.

  3. The authors also claim to release the dataset (currently only a subset is released), whose questions they manually checked. This would let others reproduce the evaluation and build on it.

Critical weaknesses
  1. Limited methodological novelty. Classifying a question and using its category to guide the response is intuitive. Checking the answer against personal history is also a natural verification step. The paper does not clearly establish the technical advance beyond combining these steps for personalization.

  2. Missing comparisons with relevant methods. The paper does not compare against simpler prompting, self-revision, or abstention methods. Useful baselines include a single prompt with the same behavioral rules, Self-Refine (Madaan et al., 2023), Chain-of-Verification (Dhuliawala et al., 2024), and self-evaluation for selective generation (Ren et al., 2023), adapted to this setting. Without these comparisons, it is hard to attribute the gains to the proposed design rather than clearer instructions or extra model calls.

  3. Incomplete analysis of tradeoffs. The method adds model calls, but latency and token costs are not reported. Some full-evidence accuracy scores also decrease. These results need closer analysis to support the claim that the method improves reliability while largely preserving personalization.

  4. Unclear role of multimodal evidence. The paper does not establish which questions require images or test performance without them. This makes it difficult to judge whether the benchmark captures challenges specific to multimodal personalization.

Questions for authors
  1. How does PRCalibrator compare with a single prompt containing the same behavioral rules, or an evidence-checking revision step? Do you have results against self-revision or abstention baselines? This would help establish whether the proposed design adds value beyond clearer instructions and extra model calls.

  2. How reliable are the judgments for open ended questions? Please clarify whether the authors’ manual checks covered only the generated questions or also the response evaluations.

  3. What proportion of questions requires visual evidence? Do you have results without images that establish the benchmark’s multimodal demands?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7j4y (1/2)

(Part 1/2)

Thank you for the thoughtful review and for recognizing that our four question categories give a useful framework for studying trustworthy personalization, and that testing with full, partial, and missing evidence goes beyond answer accuracy to show where each system fails. We also appreciate the concrete list of baselines to compare against.

[W1] Limited methodological novelty. Classifying a question to guide the response, and checking the answer against personal history, are intuitive steps. The technical advance beyond combining them for personalization is not clear.

We appreciate this comment and would like to clarify that our primary contributions lie in the benchmark and its new perspective on trustworthy personalization. While PR-Calibrator is deliberately simple and plug-and-play, its novelty goes beyond combining classification and verification. In fact, classification and history-based calibration are merely implementation steps. Instead, the core design stems from our novel analysis of trustworthy personalization, which distinguishes four question types based on their dependency on personalized evidence and corresponding trustworthy behaviors. Accordingly, PR-Calibrator is built upon this taxonomy, enabling type-specific routing and calibration to address these distinct trustworthiness requirements.

[W2/Q1] No comparison with simpler prompting, self-revision, or abstention methods, such as a single prompt with the same behavioral rules, Self-Refine, Chain-of-Verification, and self-evaluation for selective generation (Ren et al., 2023). Without them, the gains may come from clearer instructions or extra model calls.

We appreciate these concrete baseline suggestions and first note that the two ablation variants in Table 3 are the closest evidence we already have, where "+ UPD Step" puts the same rules into the generation prompt (2 calls) and "+ PC Step" is an evidence-checking revision with all handbooks (2 calls).

Variant (Qwen3.8-27B, U-Bench)CallsEPQ FGPQ FEPQ NGPQ NNQ
Base112.6728.4423.0710.3129.60
+ UPD Step (rules upfront)27.7314.4654.4755.9291.13
+ PC Step (revision only)213.0731.2143.8015.1690.93
PR-Calibrator310.8729.6964.0766.1692.87

Both variants use the same rules with one fewer call, yet neither matches the full PR-Calibrator on the None settings in Table 3. We also evaluate the four suggested baselines with the same type definitions, G1–G4, and handbooks. Multi-step baselines start from the base draft.

Each score is the macro-average of the eight Table 2 metrics.

Qwen3.8-27B, Avg of 8 cellsCallsU-BenchC-Bench
Base125.6731.52
Single prompt with the same rules144.3035.74
Self-Refine (feedback, refine)343.0135.81
CoVe (plan, verify without the draft, revise)443.3436.00
Selective generation (Ren et al., 2023)231.3632.57
PR-Calibrator350.6937.69

As shown, PR-Calibrator scores highest on both subsets. Specifically, the single prompt with the same rules stays below it, and so do Self-Refine with the same number of calls and CoVe with one more call. Therefore, the gains cannot be explained by clearer instructions or extra model calls alone.

AuthorsOpenReview ↗

Response to Reviewer 7j4y (2/2)

(Part 2/2)

[W3] Incomplete analysis of trade-offs. (1) The method adds model calls, but latency and token costs are not reported. (2) Some full-evidence accuracy scores also decrease, which needs closer analysis to support the claim that personalization is largely preserved.

  1. We agree that these costs should be reported, and we measured that prompt tokens cost 2.052.05–2.10×2.10\times the base. Moreover, UPD and drafting can run in parallel before PC. This gives 2.46×2.46\times latency for Qwen3.8-27B. Thus, the overhead is moderate, at about twice the tokens of the base assistant with no training or one-time cost. It also uses the same number of calls as Self-Refine and one fewer than CoVe (see [W2/Q1]).
  2. Personalization is preserved on average, and the few decreases do not come from over-abstention. To verify this, we compare every F-ACC cell in Table 2 before and after PR-Calibrator. The changes go in both directions and cancel out, with a mean change of only −0.01-0.01 points, whereas all 30 Partial, None, and NQ cells improve, by 22.03 points on average. Moreover, we inspect the Full responses that pass before calibration but fail after it, and only 7.1% of them abstain. Hence, PR-Calibrator does not turn overly cautious when the evidence is present.

[W4/Q3] The role of multimodal evidence is unclear. What proportion of questions requires visual evidence, and are there results without images that establish the benchmark's multimodal demands?

Thank you for this question. More than half of the U-Bench EPQs and GPQs draw on image-bearing turns, and removing the images lowers accuracy in every setting. Regarding the proportion, 177 of 300 EPQs (59%) and 165 of 289 GPQs (57%) on U-Bench link at least one evidence atom to a history turn that contains an image, and every C-Bench query comes with a new photo of the concept (Table 4). To test the multimodal demands directly, we further evaluate the Full setting of EPQ and GPQ with all history images removed and their text kept.

ModelHistoryU-Bench EPQ F-ACCU-Bench GPQ F-ACCC-Bench EPQ F-ACCC-Bench GPQ F-ACC
GPT-5.6-Solw/ images17.4747.3442.3524.62
w/o images11.6038.2030.8817.44
Qwen3.8-27Bw/ images12.6728.4446.1822.31
w/o images9.4720.1434.1214.87

As shown, F-ACC drops in all eight cells, by 3.20–12.06 points, even though the text of every turn is kept. This confirms that PersonaTrust questions cannot be answered from the text alone and do require the visual evidence.

[Q2] How reliable are the judgments for open-ended questions? Did the authors' manual checks cover only the generated questions, or also the response evaluations?

Thank you for this question. The judgments are reliable, as they stay consistent across judge families and across repeated evaluations (Appendix B.2). Specifically, judges from different model families reach 96.6% agreement on pass/fail decisions, and repeated evaluations reach 90.8% agreement (Cohen's κ=0.80\kappa = 0.80). This consistency follows from our rubric-based protocol, in which the judge only identifies observable properties of a response and the final metrics are derived by fixed rules, with no holistic judgment involved.

Regarding the manual checks, they covered the generated questions, not the response evaluations. Therefore, we will add a human–judge agreement study, in which two annotators label a sample of responses and we report Cohen's κ\kappa against the judge.

Thank you to the authors for the detailed response and the new experiments. The added baselines answer my main request. Running Self-Refine, CoVe, selective generation and a single prompt with the same rules makes the comparison much fairer. The image-removal test also helps. Accuracy drops in all eight cells, which shows the benchmark does need visual evidence.

I still have two main concerns.

First, most of the gain comes from the rules themselves. On U-Bench, the single prompt with the same rules moves the base from 25.67 to 44.30. PR-Calibrator adds 6.4 more points. On C-Bench, it is ahead of the single prompt by only 1.95 points. These results come from one model, with no variance reported. So the added value of the multi-step design over a well-written prompt looks modest.

Second, the drop in full-evidence accuracy is still not explained. The response says only 7.1% of the newly failing Full responses abstain. That means most of these failures happen for some other reason, and the response does not say what it is. Table 3 also shows that adding the rules upfront hurts Full accuracy a lot. A closer look at these cases would help support the claim that personalization is preserved.

The benchmark seems to be the strongest part of the paper. The authors also say their main contribution is the benchmark. Since only a subset is released so far, releasing the full dataset seems essential. Without it, the main contribution cannot be checked or reused.

With the lack of full benchmark release or any mentioned clafrity on the same, I will keep my current score.

Thank you for the follow-up comments. We are glad that the added baselines make the comparison much fairer and that the image-removal test confirms the benchmark's need for visual evidence. We address your remaining concerns below.

[Concern 1] Most of the gain comes from the rules. The multi-step design adds little over a single prompt with the same rules (+6.4 on U-Bench, +1.95 on C-Bench), and no variance is reported.

We agree that the rules account for a large part of the gain, but the rules are themselves part of our design, and the multi-step chain adds a further gain that is statistically significant.

  1. The rules come from our own design. We build the single-prompt baseline from the four question types, the grounding rules G1–G4, and the type-specific handbooks of PR-Calibrator, which all follow from our analysis of trustworthy personalization (Section 3 in the main paper). Therefore, this baseline is best read as a single-call variant of our method.

  2. The multi-step chain adds a clear gain on top of the same rules. We compare PR-Calibrator with this single prompt under identical rules, so their difference isolates the contribution of the multi-step design. PR-Calibrator is still 6.39 points higher on U-Bench and 1.95 points higher on C-Bench. Moreover, when the same rules enter the generation prompt ("+ UPD Step" in Table 3), GPQ F-ACC drops to 14.46, whereas PR-Calibrator keeps it at 29.69 by drafting first and calibrating afterwards (see [Concern 2]).

  3. The gain is stable across runs and statistically significant. We report both methods as mean ± sample standard deviation over five runs, with a two-sample tt-test on their difference.

    Qwen3.8-27B, Avg of 8 cellsU-BenchC-Bench
    Single prompt with the same rules44.30 ± 0.4735.74 ± 0.62
    PR-Calibrator50.69 ± 0.5237.69 ± 1.21
    pp-value<0.001< 0.0010.0125

    The gain of the multi-step design is statistically significant on both subsets (p<0.05p < 0.05).

[Concern 2] Most newly failing Full responses do not abstain, so the drop in Full accuracy remains unexplained. Table 3 also shows that adding the rules upfront hurts Full accuracy.

We apologize for leaving these cases unexplained and have now inspected them. In short, they still answer from the user's history, but the concise calibrated rewrite condenses one of the original key points, so F-ACC scores the response as zero.

  1. The remaining failures come from condensing a key point in the final calibration step. We inspect every Full response of Qwen3.8-27B on U-Bench that passes before calibration, fails after it, and does not abstain. In the final calibration step, the evidence is reorganized and the prompt asks for a concise response. At this stage, the model may condense one of the original key points of the draft, so that the point is no longer fully stated. As F-ACC is all-or-nothing (Appendix B.2), missing a single key point gives the response a score of zero, even though its personalized content is kept.
  2. These losses are offset by the responses that calibration repairs. Most Full responses that fail before calibration but pass after it contained an unsupported persona claim that the calibrated response no longer makes. The changes in F-ACC thus cancel out, with a mean change of only −0.01-0.01 points over all F-ACC cells in Table 2.
  3. Rules upfront make the model over-abstain, which is why PR-Calibrator drafts first. We compare "+ UPD Step" and PR-Calibrator on the Full questions that the base model answers correctly. With the rules in the generation prompt, the model turns conservative and abstains on 41% of them even though the evidence is present. In contrast, PR-Calibrator drafts without the rules and calibrates afterwards, so it abstains on only 5.5% of them and recovers most of the Full accuracy that "+ UPD Step" loses.

[Concern 3] Only a subset of the benchmark is released. The full dataset is essential to check and reuse the main contribution.

We fully agree that the benchmark must be available in full to be checked and reused, and we apologize that our previous response did not state our plan clearly. We commit to publicly releasing the complete PersonaTrust benchmark upon publication, as stated in the Reproducibility Statement of the paper. The release will cover all 1,682 questions with their evidence annotations and histories, the judge prompts, and all code. Meanwhile, the anonymous repository linked in the Introduction already provides sample data with the evaluation and PR-Calibrator code, so the data format and the evaluation protocol can be examined during review.

We appreciate your continued feedback and, given your recognition that our four question categories offer a useful framework for trustworthy personalization, would be grateful if you would reconsider your assessment of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.