acceptodds
Under review as a conference paper at ICLR 2027

PersonaAlign: Persona-Driven Interactive Alignment Benchmark

Abstract

Evaluating personalized large language models (LLMs) requires standardized interactive benchmarks, but existing benchmarks fail to capture the context-dependent, individualized nature of real-world personalization. To address this issue, we introduce **PersonaAlign**, a multi-turn interactive benchmark for learning from feedback. PersonaAlign uses _simulated users_—LLMs conditioned on personas—to create a diverse population of users with distinct backgrounds, writing styles, and feedback behaviors that define the preferences LLMs must learn. We instantiate PersonaAlign with a set of 1,000 personas with diverse writing style preferences, three writing tasks, and results from fourteen personalization methods (nine prompt-based, four fine-tuned, and a zero-shot baseline) across four different open-weight base models. PersonaAlign is modular and can be extended to new tasks, personas, and learning algorithms. Final results are reported with a pairwise ranking judge LLM. The simulated users and judges in PersonaAlign are validated through several human annotation studies. PersonaAlign strongly agrees (Spearman's ) with human rankings of four different personalization methods, differentiates between the methods, and shows there is room for algorithm improvement. We release our framework to support personalization research.

Primary area:
datasets and benchmarks
Keywords:
personalization, persona, benchmark, style alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kTodOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PersonaAlign evaluates writing-style personalization through repeated interactions with persona-conditioned simulated users, followed by evaluation on unseen topics without feedback. It provides 1,000 curated personas and three writing domains. The main experiments evaluate ten prompting/baseline configurations on 100 personas across four base models; four fine-tuning methods are additionally tested on OLMo-3-7B. Oracle preference prompting generally performs best, while learned methods vary by backbone. Human annotation studies assess simulator behavior, style scoring, and agreement with rankings of selected methods.

Critical strengths
  1. Separating preferred writing style from communication style allows the benchmark to vary how preferences are expressed without changing the target behavior. Retaining learned state across episodes and evaluating without further feedback then tests whether personalization transfers to new topics (§3.5). This makes the protocol useful for studying persistent adaptation.

  2. The human validation addresses several distinct components: simulator prompt selection, persona curation, style detection, and final pairwise rankings (Appendix I). The 36,000 pairwise annotations and comparison with two judge models provide meaningful evidence for the selected methods’ broad ordering, giving the evaluation more grounding than an unvalidated automated score.

  3. The comparisons reveal consequential interactions between backbone and personalization method: CoT and context compaction perform particularly well with Gemma, while Adaptive and FSE perform better with Qwen (Figure 3b). This supports evaluating algorithms across multiple backbones. Appendix C.1 further checks that rankings remain broadly stable under the tested feedback perturbations.

Critical weaknesses
  1. The human validation supports a narrower conclusion than the claimed external validity. Section 5 reports agreement with annotators applying supplied style preferences, using four deliberately well-separated methods across two models. This does not establish that learning from simulated feedback predicts success with users expressing their own preferences, or that the judge reliably distinguishes nearby leading methods. The authors acknowledge simulator limitations, but their conclusion about real-world human preferences remains stronger than the evidence. This matters because external validity is central to the benchmark’s motivation.

  2. The fine-tuning results leave evaluation robustness and checkpoint selection unresolved. Appendix D candidly reports repetition and length collapse under satisfaction-based training, then selects checkpoints with high scores and non-collapsed average length. However, the selection rule is not operationally specified, and its relationship to the three held-out evaluation tasks is unclear. The paper also does not report whether the final pairwise judge rejects these degenerate outputs. This leaves uncertainty about whether Figure 4b reflects reliable personalization gains under a reproducible selection procedure.

  3. The reported win rates need a consistent, explicit definition. Figures 3a and 12 both describe task-wise results averaged across agent LLMs, yet IF-Prompt on About Me is approximately 65% in Figure 3a and 77% in Figure 12. Figure 9 explicitly assigns half-credit to ties; the main figures do not clearly specify their convention. Different tie treatment or opponent pools could explain the discrepancy, but these must be reconciled before readers can reliably interpret performance gaps or reproduce the leaderboard.

Questions for authors
  1. Does human agreement persist for close comparisons among SF-Prompt, CC-Prompt, and Adaptive-Prompt? A focused comparison would help establish whether the benchmark distinguishes competitive methods.

  2. What exact checkpoint-selection rule was used, was selection independent of final evaluation data, and does the pairwise judge reject the degenerate outputs described in Appendix D?

  3. What numerator, denominator, tie credit, and opponent pool underlie Figures 3a, 9, and 12? Does reconciling these definitions change any method ordering?

4 comments

We thank the reviewer for their thoughtful review.

The benchmark would be stronger if it compared to humans who self-reported and annotated their own preferences.

We agree that the human validation is narrower than validating how well PersonaAlign predicts success with users expressing their own preferences. We will update the text to make this limitation and distinction more clear as well as mentioning this extension in future work. However, it was a specific design decision made to avoid non-trivial sources of noise and uncertainty in our results. The benchmark operates in the setting where a user knows their preferences and is able to use this preference information to guide their feedback. Successful preference solicitation is an active area of research and is an unsolved problem that is confounded by users not always knowing their preferences in advance. Therefore, establishing gold standard, ground truth preference for participants is not possible, making it difficult to identify if performance mismatches are due to ground truth preference solicitation protocols versus issues with the benchmark. As a first step towards a benchmark for learning from feedback, we chose a human validation setting where noise from the user was removed as a possible confounding factor.

Can the final pairwise judge catch reward-hacked results?

Yes, the final pairwise judge can catch reward-hacked results and penalize them. To answer this question, we ran the judge for the LoRA and SFT+DPO methods across various checkpoints, comparing to the checkpoint that was used in the paper. The results are below:

round 1round 3round 5round 10round 15last round
LoRA51.3%51.9%51.1%39.5%27.5%19.8%
SFT+DPO56.3%53.4%44.8%31.5%27.4%18.8%

These results show that the win rates decrease with time, as the models begin to reward hack more and more. Finally, we bucket all fine-tuned responses and label them as either repeating n-grams (loops) or non-degenerate (clean). Comparing to a zero-shot baseline, we find the following:

Output classvs all referencesvs ZeroShot
Clean50.0%73.3%
Loop11.1%23.6%

These results show that the pairwise judge is able to detect and reject degenerate outputs, as the repetitive outputs are consistently not preferred, and the more reward-hacked fine-tuning models are also consistently not preferred. We will add these investigations and results to Appendix D.

Win rates are not consistently defined

We thank the reviewer for noting this discrepancy. Win rates are computed by assigning ties half-credit. This will be clarified in the main paper and it is consistent in the code. Figure 3 in the paper used a different definition (ties as losses), but fixing this results in minimal changes to results. To verify this, we ran a correlation between the ties-as-losses results and the ties-as-half-credit results, which showed a Kendall’s tau ranking correlation of 0.96.

What checkpoint selection rule was used

Checkpoints were selected by a heuristic that checked for empty responses or repetitive n-grams. Once these appeared in any of the generations, we chose the most recent checkpoint as the “final” for evaluation purposes.

How are win-rates computed?

We assigned ties half credit, wins full credit, and losses no credit. The results ordering is unchanged between the two methods, and the win-rate calculation will be explicitly outlined in the final paper.

Thanks for the response and for running the extra experiments.

These additions help clarify the win-rate definitions and address some of my earlier questions, though I will keep my rating at 2. My main hesitation with the follow-up human study is that a single aggregate win rate (53.8%) leaves out too much detail: it is not clear whether statistical significance was tested across the 600 independent samples rather than all 6,000 clustered annotations, and there are no per-model/task breakdowns or inter-annotator agreement numbers. Also, the heuristic for checkpoint selection (checking for empty or repetitive outputs) makes sense, but you still haven't confirmed whether that check was run strictly on training/validation outputs independently of the held-out evaluation tasks.

To clarify, the statistical test for significance was carried out over all 100 personas (style sets). This aggregates much of the correlated information at a high-level (for example, repeat votes for the same sample), resulting in a more conservative statistical test. Using a paired t-test, the summarized feedback method is preferred 53.8% of the time with a significance of p=0.0016p=0.0016.

Inter-annotator agreement across all LLMs and tasks in this new study is moderate-low, with Fleiss' κ=0.157\kappa = 0.157. This reflects the close nature of these two methods, in which there is not a clear best method across samples and in which many annotators disagree about which method is superior. This finding is predicted by the PersonaAlign benchmark, which does not find the two methods to be statistically significantly different when aggregated across all models and tasks.

Results broken down by task and LLM are given in the table below:

Agent LLMTaskHuman SF winpJudge SF winp
Qwen3.5-35BAbout Me57.3%0.0039*57.5%0.039
Qwen3.5-35BBlog53.9%0.1454.5%0.23
Qwen3.5-35BProduct review55.5%0.04159.0%0.014
Gemma-4-31BAbout Me53.2%0.1852.7%0.50
Gemma-4-31BBlog51.2%0.6140.0%0.0058
Gemma-4-31BProduct review51.4%0.5849.0%0.79
Qwen3.5-35BAll tasks55.6%0.00043*57.0%0.0026*
Gemma-4-31BAll tasks51.9%0.2047.2%0.22
BothAll tasks53.8%0.0016*52.1%0.19

Where * indicates a result is significant after Bonferroni correction (α=0.059\alpha = \frac{0.05}{9}).

Finally, the checkpoint selection was conducted by checking for repeated n-grams and empty turns over all of the outputs from a training iteration, which included samples in the held-out evaluation set.

Thanks for the follow-up and the extra details. Running the paired t-test at the 100-persona level makes the significance test much cleaner, and the human–judge agreement on Qwen3.5-35B is consistent across tasks.

Still, I am keeping my rating at 2 because two core issues remain:

  1. Since checkpoint selection checked outputs that included the held-out evaluation set, the fine-tuning results in Figure 4b are affected by test-set selection rather than coming from an independent validation split.

  2. In the breakdown, the judge favors CC over SF on Gemma-4-31B blog posts (40.0% SF win rate, unadjusted p = 0.0058), whereas human annotators see no real difference (51.2%, p = 0.61). Coupled with the low inter-annotator agreement (Fleiss' κ = 0.157) and the reliance on prescribed personas rather than users' own preferences, the evidence for fine-grained and real-world reliability is still not strong enough.

Reviewer_dEL5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PersonaAlign is an interactive benchmark for evaluating how LLMs learn users’ writing preferences from feedback. It uses simulated users, evaluates personalization on held-out tasks without further feedback, and compares prompting and fine-tuning methods. The paper also includes human studies to assess the simulators and judges.

Critical strengths
  1. I think the separation between interactive learning and evaluation on new tasks is well motivated. It tests whether a method retains useful preference information beyond the current conversation.

  2. The human validation is a valuable part of the work. Together with comparisons across methods and base models, it provides useful evidence for the benchmark’s reliability.

Critical weaknesses
  1. The human ranking study asks annotators to judge compliance with predefined styles rather than their own preferences, and it focuses on methods with relatively clear performance differences. I am therefore less convinced that the reported agreement establishes reliability for real users or for comparisons between closely matched methods.

  2. The fine-tuning experiments show that satisfaction scores can improve while outputs become repetitive or overly short. Selecting checkpoints avoids some visible failures, but the paper does not clearly establish whether the final pairwise evaluation catches these failures or how sensitive the conclusions are to checkpoint selection.

Questions for authors
  1. Does the agreement with human judgments hold when comparing the strongest, closely matched methods? Is there evidence that the rankings also reflect real users’ assessments of their own preferences?

  2. Can the final pairwise judge reliably reject the degenerate outputs observed during fine-tuning? How are checkpoints selected, and would a different selection rule change the conclusions?

3 comments

We thank the reviewer for their thoughtful review.

The benchmark would be stronger if compared to humans who self-reported and annotated their own preferences.

We agree that the human validation is narrower than validating how well PersonaAlign predicts success with users expressing their own preferences. We will update the text to make this limitation and distinction more clear, including adding this extension to future work. However, it was a specific design decision made to avoid non-trivial sources of noise and uncertainty in our results. The benchmark operates in the setting where a user knows their preferences and is able to use this preference information to guide their feedback. Successful preference solicitation is an active area of research and is an unsolved problem that is confounded by users not always knowing their preferences in advance. Therefore, establishing gold standard, ground truth preference for participants is not possible, making it difficult to identify if performance mismatches are due to ground truth preference solicitation protocols versus issues with the benchmark. As a first step towards a benchmark for learning from feedback, we chose a human validation setting where noise from the user was removed as a possible confounding factor.

Does the benchmark separate closely-matched methods?

We compared four methods that were statistically significantly separated by an ANOVA and post-hoc t-tests. The other prompting based methods (such as SF, CC, Adaptive, and CoT) are not statistically significantly separated, and so we would not expect that ranking to be preserved in a human study. Our results revealed that Instruction Following (IF) is significantly better than all methods, several prompting methods (CoT, SF, Adaptive, CC, and RAG) are significantly better than Few-shot (FS), Inferred Persona (IP), Best-of-N, and Zero-Shot, and FS and IP are significantly better than Best-of-N and Zero-shot. Therefore, we selected 1 method from each “significance bucket". However, we recognize that it is important that methods which are not statistically significantly different under the benchmark are also not statistically significantly different under real human evaluation. We therefore are conducting a follow-up study comparing Summarized Feedback (SF) and Context Compaction (CC) prompting methods, and we will share results when it completes.

Can the final pairwise judge catch reward-hacked results?

Yes, the final pairwise judge can catch reward-hacked results and penalize them. To answer this question, we ran the judge for the LoRA and SFT+DPO methods across various checkpoints, comparing to the checkpoint that was used in the paper. The results are below:

round 1round 3round 5round 10round 15last round
LoRA51.3%51.9%51.1%39.5%27.5%19.8%
SFT+DPO56.3%53.4%44.8%31.5%27.4%18.8%

These results show that the win rates decrease with time, as the models begin to reward hack more and more. Finally, we bucket all fine-tuned responses and label them as either repeating n-grams (loops) or non-degenerate (clean). Comparing to a zero-shot baseline, we find the following:

Output classvs all referencesvs ZeroShot
Clean50.0%73.3%
Loop11.1%23.6%

These results show that the pairwise judge is able to detect and reject degenerate outputs, as the repetitive outputs are consistently not preferred, and the more reward-hacked fine-tuning models are also consistently not preferred. We will add these investigations and results to Appendix D.

AuthorsOpenReview ↗

Human evaluation follow-up

Following the reviewer’s suggestions that we consider more closely related methods, we conducted a human grading experiment on all three tasks (blog posts, about-me pages, and product reviews) for 200 samples per task (100 from a gemma-4-31b-it agent and 100 from a Qwen3.5-35B-A3B agent), with 10 annotations per sample (for a total of 6000 annotations) comparing Summarized Feedback (SF) with Context Compaction (CC).

Our benchmark, PersonaAlign, suggests that SF should be ranked above CC, but that the difference is not significant when aggregated across these two LLMs. Specifically, aggregating across tasks for gemma and qwen, the SF method wins 52.1% of the time, but the variance is high enough for the difference to be not-significant.

When rated by humans, the SF method wins 53.8% of the time and the difference is significant when aggregated across both models. This is an interesting finding, as it confirms the ordering of the PersonaAlign benchmark (SF > CC), but, where PersonaAlign did not predict significance, humans report a significant difference. We interpret this finding as further confirmation that the ranking of methods from PersonaAlign is reflective of a real human ranking, but the LLM judges may not be as sensitive to subtle differences as human graders are.

Thanks for the additional experiments. The SF–CC comparison and the results on repetitive outputs address some of my concern about the evaluation. I will raise my score from 1 to 2.

However, the following points still prevent me from accepting this paper. The benchmark mainly tests predefined writing styles, so I’m not sure how well the results carry over to real users whose preferences vary by task and context. Also, it’s still unclear whether the main comparisons would hold with different budgets, checkpoint choices, or user simulators. That makes it harder to tell how much of the gap comes from the methods themselves.

While I find the topic interesting, I think the paper needs much more work to meet the standards of ICLR or similar venues. I encourage the authors to continue strengthening it.

Reviewer_zuqvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PersonaAlign, an interactive benchmark for LLM writing-style personalization. Simulated users are LLMs conditioned on personas that combine four literature-grounded writing-style dimensions, a communication style and a backstory.

They give satisfaction scores and natural-language feedback over several training tasks. Agents then write held-out tasks with no feedback, and an LLM judge ranks them pairwise against each persona's style preferences.

The authors benchmark ten prompt-based conditions across four open models, plus four fine-tuned methods, and validate the final rankings with a 36k-label human study (Spearman ρ = 0.952). Personalization research currently relies on bespoke per-paper datasets, so a shared, modular and human-validated testbed like this could become a common reference point and make new methods directly comparable.

Critical strengths

The paper addresses a real, under-served need with a clean, extensible design. The stated desiderata (diversity, dynamism, robustness, separability, alignment with humans) are well chosen.

The style taxonomy builds on established linguistics work, and the modular interfaces make it easy to add tasks, personas and learning algorithms.

Critical weaknesses

The human validation could do more to show that the benchmark tells apart methods of similar quality. The ranking study uses four methods spread evenly across the leaderboard. That convincingly confirms the coarse ordering, but the benchmark's main use will be separating methods that are close together; SF, CC, Adaptive and CoT, for example, are within a few points of each other.

Extending the human comparison to a cluster of adjacent methods, and reporting judge–human agreement at the pair level, would substantially strengthen the claim that the rankings are human-validated.

Questions for authors

How well do satisfaction scores predict pairwise outcomes, at both the method level and the sample level?

Thoughts on why oracle-guided Best-of-N gains little over zero-shot?

2 comments

The results don’t try to separate methods of similar quality.

We compared four methods that were statistically significantly separated by an ANOVA and post-hoc t-tests. The other prompting based methods (such as SF, CC, Adaptive, and CoT) are not statistically significantly separated, and so we would not expect that ranking to be preserved in a human study. Our results revealed that Instruction Following (IF) is significantly better than all methods, several prompting methods (CoT, SF, Adaptive, CC, and RAG) are significantly better than Few-shot (FS), Inferred Persona (IP), Best-of-N, and Zero-Shot, and FS and IP are significantly better than Best-of-N and Zero-shot. Therefore, we selected 1 method from each “significance bucket". However, we recognize that it is important that methods which are not statistically significantly different under the benchmark are also not statistically significantly different under real human evaluation. We therefore are conducting a follow-up study comparing Summarized Feedback (SF) and Context Compaction (CC) prompting methods, and we will share results when it completes.

How well do satisfaction scores predict pairwise outcomes?

Across all 10 prompting methods, satisfaction scores are highly predictive of pairwise outcomes, with a spearman ranking correlation of 0.95 for blog posts and 0.90 for product reviews. For fine-tuning agents, correlations are weaker, at 0.40 for blog posts and 0.20 for product reviews. This distinction is because the fine-tuning methods are optimizing against the satisfaction score directly, and occasionally learn to hack in such a way that they obtain a high satisfaction score, but ultimately produce low-value responses.

Why is Best-of-N not outperforming zero-shot by more?

In our setup, the Best-of-N method uses an oracle ranker to select the best of five candidate generations, but no oracle information is used to guide the generations in the first place. Therefore, the Best-of-N method is limited by the highest quality response the generator is able to provide. Furthermore, the method does not have any memory or learning across episodes. While this method still receives feedback and can adapt within a single episode, it is stuck choosing from the best of several generic-LLM responses. Given that our evaluation setup does not allow for feedback, this means that the Best-of-N is functionally very similar to Zero-Shot in the evaluation setup — both are generating with no knowledge of the user’s style. The key difference is that Best-of-N gets several shots and can pick the best of its generations, but these may all still be quite different from the persona’s intended style.

AuthorsOpenReview ↗

Human evaluation follow-up

Following the reviewer’s suggestions that we consider more closely related methods, we conducted a human grading experiment on all three tasks (blog posts, about-me pages, and product reviews) for 200 samples per task (100 from a gemma-4-31b-it agent and 100 from a Qwen3.5-35B-A3B agent), with 10 annotations per sample (for a total of 6000 annotations) comparing Summarized Feedback (SF) with Context Compaction (CC).

Our benchmark, PersonaAlign, suggests that SF should be ranked above CC, but that the difference is not significant when aggregated across these two LLMs. Specifically, aggregating across tasks for gemma and qwen, the SF method wins 52.1% of the time, but the variance is high enough for the difference to be not-significant.

When rated by humans, the SF method wins 53.8% of the time and the difference is significant when aggregated across both models. This is an interesting finding, as it confirms the ordering of the PersonaAlign benchmark (SF > CC), but, where PersonaAlign did not predict significance, humans report a significant difference. We interpret this finding as further confirmation that the ranking of methods from PersonaAlign is reflective of a real human ranking, but the LLM judges may not be as sensitive to subtle differences as human graders are.

Comments on the paper

3 comments

We thank the reviewers for their time, consideration, and helpful reviews. Below, we would like to address three points of common feedback:

  1. The benchmark is human-validated for only four well-separated prompting methods. Why not compare more closely ranked methods to show that human rankings are preserved for fine-grained differences?

We compared four methods that were statistically significantly separated by an ANOVA and post-hoc t-tests. The other prompting based methods (such as SF, CC, Adaptive, and CoT) are not statistically significantly separated, and so we would not expect that ranking to be preserved in a human study. Our results revealed that Instruction Following (IF) is significantly better than all methods, several prompting methods (CoT, SF, Adaptive, CC, and RAG) are significantly better than Few-shot (FS), Inferred Persona (IP), Best-of-N, and Zero-Shot, and FS and IP are significantly better than Best-of-N and Zero-shot. Therefore, we selected 1 method from each “significance bucket". However, we recognize that it is important that methods which are not statistically significantly different under the benchmark are also not statistically significantly different under real human evaluation. We therefore are conducting a follow-up study comparing Summarized Feedback (SF) and Context Compaction (CC) prompting methods, and we will share results when it completes.

  1. The benchmark’s external validity would be stronger if the human study used self-reported preferences rather than predefined personas.

We agree that the human validation is narrower than validating how well PersonaAlign predicts success with users expressing their own preferences. We will update the text to make this limitation and distinction more clear as well as adding this extension to future work. However, it was a specific design decision made to avoid non-trivial sources of noise and uncertainty in our results. The benchmark operates in the setting where a user knows their preferences and is able to use this preference information to guide their feedback. Successful preference solicitation is an active area of research and is an unsolved problem that is confounded by users not always knowing their preferences in advance. Therefore, establishing gold standard, ground truth preference for participants is not possible, making it difficult to identify if performance mismatches are due to ground truth preference solicitation protocols versus issues with the benchmark. As a first step towards a benchmark for learning from feedback, we chose a human validation setting where noise from the user was removed as a possible confounding factor.

  1. Can the pairwise preference judge detect degenerate or reward-hacked outputs from the fine-tuning methods?

Yes, the final pairwise judge can catch reward-hacked results and penalize them. To answer this question, we ran the judge for the LoRA and SFT+DPO methods across various checkpoints, comparing to the checkpoint that was used in the paper. The results are below:

round 1round 3round 5round 10round 15last round
LoRA51.3%51.9%51.1%39.5%27.5%19.8%
SFT+DPO56.3%53.4%44.8%31.5%27.4%18.8%

These results show that the win rates decrease with time, as the models begin to reward hack more and more. Finally, we bucket all fine-tuned responses and label them as either repeating n-grams (loops) or non-degenerate (clean). Comparing to a zero-shot baseline, we find the following:

Output classvs all referencesvs ZeroShot
Clean50.0%73.3%
Loop11.1%23.6%

These results show that the pairwise judge is able to detect and reject degenerate outputs, as the repetitive outputs are consistently not preferred, and the more reward-hacked fine-tuning models are also consistently not preferred. We will add these investigations and results to Appendix D.

AuthorsOpenReview ↗

Human evaluation follow-up

Following the reviewer’s suggestions that we consider more closely related methods, we conducted a human grading experiment on all three tasks (blog posts, about-me pages, and product reviews) for 200 samples per task (100 from a gemma-4-31b-it agent and 100 from a Qwen3.5-35B-A3B agent), with 10 annotations per sample (for a total of 6000 annotations) comparing Summarized Feedback (SF) with Context Compaction (CC).

Our benchmark, PersonaAlign, suggests that SF should be ranked above CC, but that the difference is not significant when aggregated across these two LLMs. Specifically, aggregating across tasks for gemma and qwen, the SF method wins 52.1% of the time, but the variance is high enough for the difference to be not-significant.

When rated by humans, the SF method wins 53.8% of the time and the difference is significant when aggregated across both models. This is an interesting finding, as it confirms the ordering of the PersonaAlign benchmark (SF > CC), but, where PersonaAlign did not predict significance, humans report a significant difference. We interpret this finding as further confirmation that the ranking of methods from PersonaAlign is reflective of a real human ranking, but the LLM judges may not be as sensitive to subtle differences as human graders are.

We thank the reviewers again for their reviews and for suggesting an additional analysis to improve our work. We have conducted this additional analysis and it has lent further support to the validity of PersonaAlign, and we will add it to the final version of the paper. Additionally, we will add clarifications around how we selected checkpoints for evaluation, and how the pairwise preference judge handles reward-hacked outputs.

As the discussion period comes to a close, please let us know whether we have addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.