acceptodds
Under review as a conference paper at ICLR 2027

CURE: Constructing User-Specific Rubrics to Evaluate Multi-Turn Role-Playing Dialogue with Agent Judges

Abstract

Role-playing dialogue is an important human–AI application across virtual companionship, narrative play, and social interaction, where users increasingly engage with characters in long-running conversations spanning many turns. These long interactions provide contexts that reveal users' evolving goals, preferences, and boundaries. They impose greater demands on role-playing models and evaluation methods: models must maintain character consistency while adapting over time, and evaluators must assess interactions against user-specific requirements. However, existing evaluation still centers largely on isolated responses or fixed single-turn probes. We therefore aim to build a multi-turn evaluation framework for role-playing dialogue. We first observe that multi-turn role-playing evaluation lacks a dedicated benchmark dataset. Using an emotional BDI mechanism, we construct and release EmoRoleBench, a benchmark of 1,751 stateful dialogues across 22 characters, 16 user profiles, and five role-playing models. Each dialogue has an independently generated post-session satisfaction label that is hidden from the evaluator. We then introduce CURE, a construct-first Agent-Judge framework that derives an explicit user-specific rubric from preferences and boundaries expressed across turns, corrects it with persona-grounded anchors, and gathers turn-level and temporal evidence before scoring. On the predefined test split, CURE achieves a Spearman correlation of 0.678 with simulated-user satisfaction, compared with 0.533 for the strongest external baseline. It also performs better in matched dialogue comparisons, remains stable across three Judge backbones, and shows positive alignment with persona-conditioned human ratings. These results show that CURE more effectively captures user preferences that emerge across turns and translates them into user-aligned evaluations of role-playing dialogue. Code and data are available at https://anonymous.4open.science/r/cure-review-artifact-2026-1E3C/.

Primary area:
datasets and benchmarks
Keywords:
role playing, LLM-as-a-Judge, Agent-as-a-Judge, multi-turn dialogue evaluation, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UktpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CURE frames subjective multi-turn role-playing evaluation as a criterion-construction problem. Given a completed public dialogue and a character card, its Construct stage infers user-specific requirements from quoted user utterances, Feedback audits the draft rubric against character-grounded anchors and explicit boundaries, and Score gathers dialogue and temporal evidence to evaluate the resulting fixed rubric. The authors also introduce EmoRoleBench, an emotional BDI-based user-simulation benchmark containing 1,751 released dialogues, while describing an original corpus of 1,991 dialogues across more character cards. A separate LLM generates a post-session satisfaction label from the private user profile and simulator state, which is hidden from CURE during evaluation. On the authors’ reported frozen test run, CURE obtains Spearman rho = 0.678 versus 0.533 for DynSess-Eval, achieves 77.73% agreement in matched cross-model comparisons, and shows positive but limited alignment with 78 human ratings. The evidence supports improved agreement with the particular simulated satisfaction signal, but does not yet establish alignment with independent real-user satisfaction.

Critical strengths
  1. The construct-before-score decomposition is conceptually clear and practically useful. CURE separates the question of what matters for the current user from the later question of how well the dialogue satisfies those criteria. The use of quoted user evidence, persona-grounded correction, locked rubrics, and message-linked scoring evidence addresses a genuine limitation of fixed or implicit LLM-judge rubrics.

  2. The empirical package is broader and more transparent than is typical for LLM-evaluation work. The paper includes a long-horizon simulated corpus, multiple external baselines, rubric ablations, matched cross-model comparisons, dialogue-length and profile analyses, cross-backbone diagnostics, and an illustrative boundary-violation case. The inclusion of negative subgroup results and implementation details makes the proposed evaluator easier to inspect critically.

Critical weaknesses
  1. The principal validation target is not independent enough to support the paper’s strongest claims. The satisfaction label S is generated by DeepSeek V4 Flash using the private user profile, BDI trajectory, satisfaction/dealbreaker signals, termination-related information, and auxiliary LLM judgments, while the main CURE judge also uses DeepSeek V4 Flash. The observed rho = 0.678 may therefore reflect shared model-family behavior, simulator-specific shortcuts, or visible transcript correlates of exit behavior rather than genuine alignment with human satisfaction. The human check contains only 78 non-random dialogues, has very low inter-rater reliability (ordinal Krippendorff’s alpha = .168; ICC(2,1) = .243), and the CURE–OpenJudge difference is only +.044 with a confidence interval spanning zero. The within-profile mean gain over DynSess-Eval is also only .032 with a 95% interval of [-.031, .097], while profiles A4 and A6 show negative gains. Thus the pooled improvement does not convincingly establish that personalized rubric construction improves within-user evaluation.

  2. Reproducibility and attribution are materially underdeveloped. The manuscript reports inconsistent corpus and split counts: 1,991 original dialogues versus 1,751 released dialogues, a 1,271-dialogue test split, a 1,270-dialogue all-method comparison, and a released Table 13 test partition of 1,111 dialogues. Three character cards and 240 dialogues are withheld, and the human-rating analysis refers to 24 characters although the anonymous release contains 22. The main method is also largely hand-designed prompt/tool orchestration, but there are no held-out-character or held-out-profile tests, no repeated simulator seeds, no refusal- or length-matched baseline, and no stage-level or call/token-budget ablations. CURE uses 78.9 model calls per dialogue, while several baselines use only 2–4 calls, and token cost and wall-clock cost are not reported. Finally, the complete Construct and Score tool traces are not persisted, which weakens the claim that the entire evaluation path is auditable.

Questions for authors
  1. Clarification on the corpus and partitions: Please provide an exact mapping between the original 1,991-dialogue corpus and the 1,751-dialogue anonymous release, including the identities and partition membership of the withheld 240 dialogues. How do the reported 1,271 test dialogues, the 1,270 dialogues with predictions from all methods, and the 1,111 test dialogues in Table 13 correspond to one another? Can every number in Table 1 and Table 3 be recomputed from the anonymous release alone?

  2. Independent validity of the reference target: What happens when the satisfaction target is generated by an independent model family, by multiple independent labelers, or by a non-LLM/rule-based procedure? Can the authors provide a statistically powered human comparison between CURE and OpenJudge on a random or stratified sample? If not, please limit the claims explicitly to alignment with the simulator-generated label and explain how shared DeepSeek behavior is ruled out.

  3. Information-set and generalization clarification: Appendix A.4 states that human annotators saw a public user persona, character card, and complete dialogue, whereas the main evaluation description says that CURE sees only the public dialogue and character card. What exactly is contained in the public user persona, and was it available to all automated judges? In addition, do the conclusions hold under held-out character, held-out profile, and held-out target-model splits, or under repeated simulator seeds?

  4. Attribution and reproducibility: How much of the gain remains under matched call, token, or wall-clock budgets? Please isolate Construct, Feedback, Score, evidence tools, gated items, and the weighted mechanical aggregate through ablations. Can the authors release byte-exact prompts, model/API versions, failure rates, repeated-call variance, and complete tool traces so that the claimed auditability can be independently checked?

2 comments

We thank the reviewer for the careful assessment and constructive questions.

Q1. Corpus, partitions, and reproducibility

We apologize for the unclear distinction between the original corpus and the anonymous release. The release excludes dialogues from three confidential character cards.

Data scopeAll dialoguesTest dialogues
Original corpus1,9911,271
Excluded240160
Anonymous release (Table 13)1,7511,111

The main comparison uses the 1,270 original test dialogues with predictions from every method. The separately executed ablations cover all 1,271. Consequently, not every full-corpus result in Tables 1 and 3 can be recomputed from the anonymous release alone. We will specify the data scope and evaluation set for each result in the manuscript and release documentation.

Q2. Independent validity of the satisfaction reference

GLM-5.2 generates user utterances and BDI updates. DeepSeek V4 Flash generates the satisfaction reference S. Tables 4 and 7 report cross-backbone diagnostics. The correlations with mean human ratings H use the same 76 dialogues across judges.

JudgeValid dev. nρ with Sρ with H
DeepSeek V4 Flash1980.7030.577
Qwen3.8 Flash1960.6430.482
GLM-5.3 Flash2000.6470.572

Qwen differs from both the simulator and labeler families, and its correlation with H has a 95% CI of [0.250, 0.652]. These checks retain S. Relabeling with another model family remains untested.

We also conducted validation with three independent human annotators, blinded to S and evaluator predictions. On the 78 dialogues rated by all three, ρ(S, H) = 0.642. CURE's alignment remains consistent across aggregation choices, including standardization within rater (Appendix A.4).

Human aggregationMeanMedianStandardized mean
ρ with CURE0.5580.5580.562

The 78 dialogues were the non-random overlap of the three annotators’ completed ratings. On this exploratory sample, CURE had the highest observed correlation with human ratings among the evaluators (ρ = 0.558), followed by OpenJudge (ρ = 0.514). The paired difference was +0.044 (95% CI [−0.181, 0.238]).

Q3. Public inputs and generalization

Human annotators and automated evaluators received the same public character information and complete dialogue, including the character card, trait hierarchy, facet list, and dialogue turns. “Public user persona” is imprecise wording and will be corrected in Section 4.4 and Appendix A.4. The persona field describes the role-playing character. Private user profiles, BDI states, termination metadata, and reference labels were hidden. Evaluator predictions were also hidden from annotators.

The partitions are disjoint at the dialogue level. Across five role-playing models and three judge backbones, all 15 within-model correlations are positive (ρ from 0.302 to 0.770), supporting consistency across the evaluated models.

Q4. Workflow, performance, and reproducibility

Construct defines criteria, Feedback audits them, and Score applies the locked rubric. These three stages form a sequential workflow. Removing Construct or Score would remove the criteria or evaluation output. Our ablations examine design choices within this workflow.

Table 1(a) shows improvements over DynSess-Eval across all four metrics. A lower MAE indicates smaller error.

MethodSpearman's ρKendall's tau-bMAEW@1
CURE0.6780.5550.97568.8%
DynSess-Eval0.5330.4261.18551.2%

Table 1(b) evaluates rubric design choices.

VariantSpearman's ρ
Full CURE0.678
Generic rubric0.537
No gated items0.626
No custom M70.608
Personalized base levels0.610

Table 9 reports the mean number of model calls per dialogue.

CUREG-EvalOpenJudge
78.980.0135.3

Prompts require user quotations for criteria, character compatibility checks, and message-linked scoring evidence. M1–M6 retain fixed levels, and Feedback allows two audits and one revision. The final score is holistic. The weighted aggregate is diagnostic (ρ = 0.948 with the holistic score).

We will expand the appendix with exact prompt text, available model/API and decoding configurations, tool schemas, and scoring/fallback rules.

Thank you for the clarifications. I consider the corpus counts reconciled, and your explanation resolves my concern about different information being provided to human annotators and automated evaluators, subject to the corresponding manuscript corrections. I also acknowledge that the cross-family judge results provide useful evidence against an explanation based solely on the shared DeepSeek backbone.

My remaining concerns are primarily about validation and attribution. The human comparison remains exploratory and does not establish an advantage over OpenJudge, while the reference labels remain unchanged across judge backbones. The existing rubric ablations are informative, but the workflow explanation does not isolate the incremental value of Feedback or holistic scoring. Likewise, the reported call counts do not resolve the performance–budget tradeoff relative to DynSess-Eval.

I appreciate your acknowledgment of the release limitations and commitment to documenting the exact evaluation scope and implementation. However, the remaining evidential gaps leave my overall recommendation at weak rejection.

Reviewer_GKHtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CURE, an agent-judge framework for evaluating multi-turn role-playing dialogue. Instead of using a fixed rubric, CURE constructs a dialogue-specific rubric based on the user's turns in a role-playing dialog. It then audits this rubric against persona-derived anchors. Lastly, a tool-using agent scores the dialogue against the obtained rubric.

Authors also build an emotional Belief–Desire–Intention user simulator and use it to construct EmoRoleBench, which contains 1751 simulated dialogues across 22 characters, 16 user profiles and five role-playing models. Each dialogue also has an LLM-generated post-session satisfaction label. On the test split, CURE reaches a Spearman correlation of 0.678 with this label, compared with 0.533 for a baseline (DynSess-Eval). Authors also conducted some human rating study.

Critical strengths
  • The idea is intuitive and reasonable. A fixed rubrics cannot capture the varying preferences of different users in role-playing scenarios.

  • The constructed benchmark is a valuable resource.

Critical weaknesses
  • Limited novelty. There are a plenty of benchmarks with instance-specific rubrics (e.g., BiGGen Bench), and agent-as-a-judge is commonly applied. As a result, the movitivation and novelty of CURE could be narrowed to applying/generating the instance-specific rubrics, for the role-playing scenario. Authors should clearly discuss what is the difference w.r.t. the instance-specific rubric literature.

  • I feel there is a circular labeling and the evaluating. I.e., the labels and the users' turns are both simulated based on the user profiles, using DeepSeekV4Flash, so it is somehow expected CURE achieves better results than other baselines.

  • Missing baseline details. For example, it is unclear how the baseline outputs are converted to a "1–5 scale under a fixed protocol-specific rule" in line364. In general, I would recommend users to clearly describe the baseline configurations to make sure the comparisons are fair.

  • I think some ablations are needed to comfirm the intermediate steps are correct/effective. E.g., experiments without the Feedback steps etc.

  • Even though human study is conducted, the inter-annotator agreement is pretty low (Krippendorff’s alpha 0.168), it is unclear if the human study can validate the method or not.

  • Limited scope and language. In the main text, there is no descriptions about the dataset properties (e.g., a Chinese dataset).

Questions for authors
  • The reported results use a 1271-dialogue test split (line357), but the Table 13 shows the test split contains 1111 dialogues. Which is the correct number?

  • How were the baseline outputs mapped to the 1–5 scale?

1 comment

We thank the reviewer for the detailed feedback and recognition of CURE's motivation and EmoRoleBench's value.

W1. Novelty and instance-specific rubrics

In multi-turn role-playing, user preferences vary substantially and emerge through interaction, so the evaluation criteria themselves must be inferred. Our workflow grounds each criterion in user quotations, checks it against character constraints, and locks the rubric before scoring.

BiGGen Bench supplies instance-specific criteria with benchmark instances. CURE constructs criteria from the current dialogue. In Table 1(b), Spearman's ρ rises from 0.537 with a generic character-grounded rubric to 0.678 with CURE. We will clarify this distinction in related work.

W2. Circular labeling and evaluation

GLM-5.2 generates user turns and BDI updates. DeepSeek V4 Flash generates simulated satisfaction S. All main judges use DeepSeek and the same public inputs. Private simulator data and S are hidden.

H is the mean rating from three independent annotators blinded to S and judge outputs (Tables 3 and 7, Appendix A.4).

ScoreReferencenSpearman's ρ
CURE (main)H780.558
SH780.642
CURE (Qwen)S1960.643
CURE (Qwen)H760.482

Qwen3.8 Flash differs from both simulator and labeler families. Its S result uses development dialogues. Its H result uses the common subset (95% CI [0.250, 0.652]). Across three judges and five models, all 15 within-model correlations were positive (0.302–0.770), with Kendall's W = 0.956 for model rankings. These results support CURE's generalizability across judge backbones and alignment with independent human judgments.

W3. Baseline configurations and score conversion

All baselines use the same public inputs and DeepSeek V4 Flash. Fixed method-specific rules convert native scores to 1–5. We use the following endpoint mapping for a native 1–100 score.

s5=1+4(s100−1)99s_5 = 1 + \frac{4(s_{100} - 1)}{99}

This conversion preserves score order. The endpoint mapping above applies to native scores on the 1–100 scale. The anonymous release omits the baseline-specific aggregation and conversion code. We will document the original outputs and conversion settings that we can verify for each method.

W4. Feedback and ablation evidence

Feedback audits user-specific criteria, weights, gates, and M7 against user quotations and character constraints. It permits at most two audits and one targeted revision. Accepted items are locked. Unresolved gates become normal, unresolved base items revert to base criteria, and unsupported M7 is removed.

Construct–Feedback–Score is a coupled workflow. Removing Feedback changes Score's rubric and context, requiring revised inputs, outputs, prompts, and fallback rules.

Table 1(b) reports the rubric ablations.

VariantDesign examinedSpearman's ρ
Full CUREReference0.678
Generic rubricUser-specific criteria0.537
No gated itemsExplicit boundaries0.626
No custom M7Needs beyond M1–M60.608
Personalized levelsShared scoring anchors0.610

For the last three variants, the 95% CIs for Δρ (variant minus full CURE) are below zero (Appendix A.3). Together with W2's cross-backbone and human validation, they support CURE's robustness and generalizability.

W5. Human-rating agreement

Role-playing users have highly diverse narrative goals, emotional needs, and relationship expectations, making quality judgments strongly user-dependent. Proactive plot advancement may delight one user but undermine another's agency. The same behavior can therefore receive legitimately different ratings.

Our three annotators bring different user perspectives to the shared rating protocol. Their disagreement (α = 0.168) is consistent with this preference heterogeneity. CURE aims to capture shared preferences while respecting individual needs. CURE's human correlations are consistent across mean (0.558), median (0.558), and within-rater-standardized aggregation (0.562) in Appendix A.4.

W6. Dataset language and scope

We will move this appendix summary into the main text. The release covers 1,751 Chinese dialogues, 22 characters, 16 user profiles, and five role-playing models. The prompts are also in Chinese.

Length (exchanges)1–56–1516–3031–60
Share24.5%45.9%4.1%25.5%
Exit reasonDealbreakerLow engagementReply limit
Share38.8%40.2%21.0%

Q1. Test-set counts

The original corpus has 1,991 dialogues and 1,271 test cases. Excluding 240 dialogues from three confidential cards (160 test cases) leaves 1,751 released dialogues and 1,111 test cases (Table 13). Main results use 1,270 original test cases with complete predictions. Separate ablations cover all 1,271. We will label these scopes in the main text and captions.

Reviewer_qJ3kOpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CURE, an agent judge that evaluates multi-turn role-playing dialogues by extracting user-grounded rubrics, enforcing character constraints, and retrieving interaction evidence for scoring. The authors further release EmoRoleBench, a stateful BDI-based benchmark with hidden user satisfaction targets.

Critical strengths

The paper addresses a relevant problem and proposes a well-motivated framework for personalized dialogue evaluation. The experimental results are encouraging, with comprehensive baseline comparisons, ablation studies, and human evaluations supporting the effectiveness of the approach.

Critical weaknesses

1.The layout of the tables in the main text is irregular, resulting in a significant waste of space. Furthermore, Table 2 contains both tables and figures.

2.All evaluation splits share the identical synthetic generator and prompting template without any out of distribution testing on external benchmarks or real human logs Consequently CURE might merely overfit this specific simulator style rather than demonstrating genuine generalizability across diverse unconstrained real world dialogue environments.

Questions for authors

I suggest the authors adjust the table format and select other publicly available dialogue data (not necessarily the complete benchmark dataset, but a portion of open-source data) for testing, so that this method can be compared with other methods.

2 comments

We thank the reviewer for the careful reading and for recognizing the motivation of CURE and the breadth of our evaluation. We appreciate the suggestions on presentation and external validation.

W1. Table presentation

We agree that the table layout can be improved. We will standardize column alignment, spacing, and caption formatting. For Table 2, we will present the dialogue illustration as a figure and the evaluator scores as a separate table, with explicit cross-references. This will make the evidence and numerical comparison easier to follow.

W2. Generalization and external dialogue data

We appreciate this suggestion. Role-playing satisfaction is highly subjective, and benchmarks differ in their evaluation units and criteria. Our target is satisfaction over a complete session, conditioned on the preferences and boundaries expressed by its user. The external benchmarks we examined do not directly match this target and labeling protocol. This motivated EmoRoleBench.

Informed by our analysis of real-user needs, EmoRoleBench includes 16 profiles covering storytelling, companionship, casual interaction, critical testing, and support seeking. Its BDI simulator models evolving goals, emotional reactions, and departure conditions to reflect realistic interaction dynamics (Section 3.1 and Appendix D.1).

Our existing experiments compare CURE and DynSess-Eval against simulated satisfaction S. We report Spearman's ρ for individual dialogues and ranking agreement for pairs with the same character and user profile.

SettingnMetricCUREDynSess-Eval
Main test1,270ρ0.6780.533
Matched pairs1,026Agreement77.73%74.90%
60 character replies272ρ0.4930.344

Across three judge backbones and five role-playing models, all 15 within-model correlations were positive.

ValidationMetricResult
Within-model correlationsRange of ρ0.302–0.770
Model rankingsKendall's W0.956
CURE vs. humanρ0.558
Simulated vs. humanρ0.642

These results are reported in Tables 1, 3, and 7 and Appendices A.4–A.6. Three independent annotators rated 78 dialogues, blinded to simulated satisfaction S and evaluator outputs. CURE's human alignment also remains consistent under median and within-rater-standardized aggregation (ρ = 0.558 and ρ = 0.562, respectively).

Together, the user-informed benchmark design, consistency across models and dialogue lengths, and agreement with independent human judgments support CURE's applicability to realistic role-playing evaluation. We will clarify the profile-construction rationale and these validation findings in the revision.

Thank you for the response. However, all evaluations, including human evaluation, remain based on simulator-generated dialogues from EmoRoleBench. No external dialogue datasets or real-world conversation logs have been used to evaluate CURE. Therefore, its generalization to diverse dialogue sources and real-world settings remains unverified. I will maintain my original rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.