- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CURE frames subjective multi-turn role-playing evaluation as a criterion-construction problem. Given a completed public dialogue and a character card, its Construct stage infers user-specific requirements from quoted user utterances, Feedback audits the draft rubric against character-grounded anchors and explicit boundaries, and Score gathers dialogue and temporal evidence to evaluate the resulting fixed rubric. The authors also introduce EmoRoleBench, an emotional BDI-based user-simulation benchmark containing 1,751 released dialogues, while describing an original corpus of 1,991 dialogues across more character cards. A separate LLM generates a post-session satisfaction label from the private user profile and simulator state, which is hidden from CURE during evaluation. On the authors’ reported frozen test run, CURE obtains Spearman rho = 0.678 versus 0.533 for DynSess-Eval, achieves 77.73% agreement in matched cross-model comparisons, and shows positive but limited alignment with 78 human ratings. The evidence supports improved agreement with the particular simulated satisfaction signal, but does not yet establish alignment with independent real-user satisfaction.
Critical strengths
-
The construct-before-score decomposition is conceptually clear and practically useful. CURE separates the question of what matters for the current user from the later question of how well the dialogue satisfies those criteria. The use of quoted user evidence, persona-grounded correction, locked rubrics, and message-linked scoring evidence addresses a genuine limitation of fixed or implicit LLM-judge rubrics.
-
The empirical package is broader and more transparent than is typical for LLM-evaluation work. The paper includes a long-horizon simulated corpus, multiple external baselines, rubric ablations, matched cross-model comparisons, dialogue-length and profile analyses, cross-backbone diagnostics, and an illustrative boundary-violation case. The inclusion of negative subgroup results and implementation details makes the proposed evaluator easier to inspect critically.
Critical weaknesses
-
The principal validation target is not independent enough to support the paper’s strongest claims. The satisfaction label S is generated by DeepSeek V4 Flash using the private user profile, BDI trajectory, satisfaction/dealbreaker signals, termination-related information, and auxiliary LLM judgments, while the main CURE judge also uses DeepSeek V4 Flash. The observed rho = 0.678 may therefore reflect shared model-family behavior, simulator-specific shortcuts, or visible transcript correlates of exit behavior rather than genuine alignment with human satisfaction. The human check contains only 78 non-random dialogues, has very low inter-rater reliability (ordinal Krippendorff’s alpha = .168; ICC(2,1) = .243), and the CURE–OpenJudge difference is only +.044 with a confidence interval spanning zero. The within-profile mean gain over DynSess-Eval is also only .032 with a 95% interval of [-.031, .097], while profiles A4 and A6 show negative gains. Thus the pooled improvement does not convincingly establish that personalized rubric construction improves within-user evaluation.
-
Reproducibility and attribution are materially underdeveloped. The manuscript reports inconsistent corpus and split counts: 1,991 original dialogues versus 1,751 released dialogues, a 1,271-dialogue test split, a 1,270-dialogue all-method comparison, and a released Table 13 test partition of 1,111 dialogues. Three character cards and 240 dialogues are withheld, and the human-rating analysis refers to 24 characters although the anonymous release contains 22. The main method is also largely hand-designed prompt/tool orchestration, but there are no held-out-character or held-out-profile tests, no repeated simulator seeds, no refusal- or length-matched baseline, and no stage-level or call/token-budget ablations. CURE uses 78.9 model calls per dialogue, while several baselines use only 2–4 calls, and token cost and wall-clock cost are not reported. Finally, the complete Construct and Score tool traces are not persisted, which weakens the claim that the entire evaluation path is auditable.
Questions for authors
-
Clarification on the corpus and partitions: Please provide an exact mapping between the original 1,991-dialogue corpus and the 1,751-dialogue anonymous release, including the identities and partition membership of the withheld 240 dialogues. How do the reported 1,271 test dialogues, the 1,270 dialogues with predictions from all methods, and the 1,111 test dialogues in Table 13 correspond to one another? Can every number in Table 1 and Table 3 be recomputed from the anonymous release alone?
-
Independent validity of the reference target: What happens when the satisfaction target is generated by an independent model family, by multiple independent labelers, or by a non-LLM/rule-based procedure? Can the authors provide a statistically powered human comparison between CURE and OpenJudge on a random or stratified sample? If not, please limit the claims explicitly to alignment with the simulator-generated label and explain how shared DeepSeek behavior is ruled out.
-
Information-set and generalization clarification: Appendix A.4 states that human annotators saw a public user persona, character card, and complete dialogue, whereas the main evaluation description says that CURE sees only the public dialogue and character card. What exactly is contained in the public user persona, and was it available to all automated judges? In addition, do the conclusions hold under held-out character, held-out profile, and held-out target-model splits, or under repeated simulator seeds?
-
Attribution and reproducibility: How much of the gain remains under matched call, token, or wall-clock budgets? Please isolate Construct, Feedback, Score, evidence tools, gated items, and the weighted mechanical aggregate through ablations. Can the authors release byte-exact prompts, model/API versions, failure rates, repeated-call variance, and complete tool traces so that the claimed auditability can be independently checked?
2 comments
Thank you for the clarifications. I consider the corpus counts reconciled, and your explanation resolves my concern about different information being provided to human annotators and automated evaluators, subject to the corresponding manuscript corrections. I also acknowledge that the cross-family judge results provide useful evidence against an explanation based solely on the shared DeepSeek backbone.
My remaining concerns are primarily about validation and attribution. The human comparison remains exploratory and does not establish an advantage over OpenJudge, while the reference labels remain unchanged across judge backbones. The existing rubric ablations are informative, but the workflow explanation does not isolate the incremental value of Feedback or holistic scoring. Likewise, the reported call counts do not resolve the performance–budget tradeoff relative to DynSess-Eval.
I appreciate your acknowledgment of the release limitations and commitment to documenting the exact evaluation scope and implementation. However, the remaining evidential gaps leave my overall recommendation at weak rejection.
We thank the reviewer for the careful assessment and constructive questions.
Q1. Corpus, partitions, and reproducibility
We apologize for the unclear distinction between the original corpus and the anonymous release. The release excludes dialogues from three confidential character cards.
The main comparison uses the 1,270 original test dialogues with predictions from every method. The separately executed ablations cover all 1,271. Consequently, not every full-corpus result in Tables 1 and 3 can be recomputed from the anonymous release alone. We will specify the data scope and evaluation set for each result in the manuscript and release documentation.
Q2. Independent validity of the satisfaction reference
GLM-5.2 generates user utterances and BDI updates. DeepSeek V4 Flash generates the satisfaction reference S. Tables 4 and 7 report cross-backbone diagnostics. The correlations with mean human ratings H use the same 76 dialogues across judges.
Qwen differs from both the simulator and labeler families, and its correlation with H has a 95% CI of [0.250, 0.652]. These checks retain S. Relabeling with another model family remains untested.
We also conducted validation with three independent human annotators, blinded to S and evaluator predictions. On the 78 dialogues rated by all three, ρ(S, H) = 0.642. CURE's alignment remains consistent across aggregation choices, including standardization within rater (Appendix A.4).
The 78 dialogues were the non-random overlap of the three annotators’ completed ratings. On this exploratory sample, CURE had the highest observed correlation with human ratings among the evaluators (ρ = 0.558), followed by OpenJudge (ρ = 0.514). The paired difference was +0.044 (95% CI [−0.181, 0.238]).
Q3. Public inputs and generalization
Human annotators and automated evaluators received the same public character information and complete dialogue, including the character card, trait hierarchy, facet list, and dialogue turns. “Public user persona” is imprecise wording and will be corrected in Section 4.4 and Appendix A.4. The persona field describes the role-playing character. Private user profiles, BDI states, termination metadata, and reference labels were hidden. Evaluator predictions were also hidden from annotators.
The partitions are disjoint at the dialogue level. Across five role-playing models and three judge backbones, all 15 within-model correlations are positive (ρ from 0.302 to 0.770), supporting consistency across the evaluated models.
Q4. Workflow, performance, and reproducibility
Construct defines criteria, Feedback audits them, and Score applies the locked rubric. These three stages form a sequential workflow. Removing Construct or Score would remove the criteria or evaluation output. Our ablations examine design choices within this workflow.
Table 1(a) shows improvements over DynSess-Eval across all four metrics. A lower MAE indicates smaller error.
Table 1(b) evaluates rubric design choices.
Table 9 reports the mean number of model calls per dialogue.
Prompts require user quotations for criteria, character compatibility checks, and message-linked scoring evidence. M1–M6 retain fixed levels, and Feedback allows two audits and one revision. The final score is holistic. The weighted aggregate is diagnostic (ρ = 0.948 with the holistic score).
We will expand the appendix with exact prompt text, available model/API and decoding configurations, tool schemas, and scoring/fallback rules.