acceptodds
Under review as a conference paper at ICLR 2027

Capability Calibration in Role-Playing Language Models: A Benchmark and Metacognitive Agentic Framework

Abstract

Research on role-playing large language models (LLMs) has largely emphasized fidelity to the identities, personalities, and behaviors of characters and personas. For professional roles across career stages, however, a key challenge remains underexplored: providing useful assistance within the expertise and decision authority supported by a persona's career history—a requirement we term career-grounded capability calibration. LLMs may draw on capabilities beyond those supported by the assigned profile, causing junior or unrelated personas to respond like experts. We introduce CapRoleBench, which holds 355 workplace tasks fixed while varying career evidence across Senior, Mid, Junior, and Unrelated profiles to evaluate response fidelity and capability ordering. We further propose CapRoleAgent (CRA), a training-free, model-agnostic metacognitive framework that aligns career-supported capabilities with task requirements, enforces behavioral contracts, and audits overreach and omission. Across 14 open and proprietary LLMs, we observe systematic capability leakage under direct career-profile prompting. On a matched 100-task subset, CRA raises macro-average Ordering Consistency from 65.9% to 81.6% and lowers macro-average Overreach Rate from 32.0% to 10.9%, with both metrics improving for all 14 models. Profile-aware evaluation further shows higher Knowledge Fidelity and Behavior Coherence with fewer omissions, indicating improved role fidelity while preserving useful assistance. Evaluations by three LLM judges and persona-blind human raters corroborate the gains in capability ordering. The code is available at https://anonymous.4open.science/r/CapRoleAgent.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Role-Playing, Capability Calibration, Metacognition, Agentic Framework, LLM Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wgE9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies career-grounded capability calibration in role-playing language models. The goal is for a model to express expertise, execution scope, and decision authority supported by a persona’s career history, rather than using its general capabilities to answer every task as an expert. The authors introduce CapRoleBench, which fixes 355 workplace tasks and varies four career profiles—Senior, Mid, Junior, and Unrelated—yielding 1,420 controlled task–profile instances. They also propose CapRoleAgent (CRA), a training-free inference-time framework that constructs a capability graph from the career history, monitors task requirements and evidence alignment, selects a behavioral regime, generates a response, and performs feedback-based revision.

CPP and CRA are compared on a fixed 100-task diagnostic subset across 14 language models. CRA improves macro-average Ordering Consistency from 65.9% to 81.6% and reduces Overreach Rate from 32.0% to 10.9%. The paper also reports profile-aware LLM-judge scores, leave-one-component-out ablations, results from three evaluators, and a small persona-blind human ranking study. The reported improvements are strongest for the Monitoring and Control components.

Critical strengths
  1. Controlled benchmark framing. CapRoleBench fixes the workplace task while varying only the career evidence, which isolates a meaningful distinction between solving a task and expressing capability that is appropriate for a persona. This is a useful problem formulation for role-playing and agent evaluation.

  2. Broad empirical coverage. The paper evaluates 14 language models, reports multiple automated evaluators, component ablations, and a persona-blind human ranking study. The improvement in ordering metrics is directionally consistent across the evaluated models and evaluators, which makes the phenomenon worth further investigation even though the current validation has important limitations.

Critical weaknesses
  1. The central CRA evaluation is vulnerable to benchmark-selection and information-leakage effects. CRA is tested only on a fixed 100-task diagnostic subset selected using criteria that include ordering difficulty, a useful low-evidence boundary, and clear Senior support. No matched results are provided for the remaining 255 tasks or for an independently held-out random test set. Moreover, the Senior/Mid/Junior ordering is created through task-specific auditing and relabeling of slices from the same career trajectory, while Unrelated is defined using a different occupation group. The CRA controller is then described as using the current occupation’s relation to the target occupation and ESCO code, although that target metadata is absent from the shared CPP prompt. If this metadata is supplied to CRA, the comparison is not information matched; if it is inferred from the task, the exact interface is unclear. The observed OC and OVR gains may therefore reflect matching the benchmark’s construction rules rather than general capability calibration.

  2. The causal attribution and practical utility of CRA are underdetermined. CRA uses approximately 7.05 times the tokens and 4.90 times the cost of CPP, multiple LLM-generated intermediate states, and up to two self-revisions, but there is no equal-budget baseline using generic self-critique, longer prompting, structured career prompts, retrieval, or explicit refusal/authority control. The same general type of LLM judges the profile-blind ordering and profile-aware metrics, and GPT-4.1 is both the primary evaluator and one of the evaluated backbones. No objective expert labels for task correctness, supported authority, or practical helpfulness are reported, and no confidence intervals, paired statistical tests, or repeated generations are provided. The large increase in Refusal Rate is not quality-sensitive, and Behavior Coherence or Omission Rate worsens for Llama3.1-8B and Qwen3-8B. Thus higher ordering consistency may partly be achieved by more cautious or refusal-heavy behavior rather than by better useful assistance within the supported capability boundary.

Questions for authors
  1. Critical clarification: Does CRA receive the task’s target occupation, ESCO code, or any hidden benchmark metadata when selecting the behavioral regime? The shared CPP prompt does not contain this information, while Section B.3 says that the controller classifies the current occupation relative to the target occupation. If the metadata is supplied, can the authors rerun the comparison with identical information available to CPP? If it is inferred from the task, please provide the exact prompt and interface.

  2. Material validation question: How were the 100 CapRoleBench-100 tasks selected, and were the selection criteria fixed before observing CPP or CRA outputs? What are CRA’s results on the remaining 255 tasks, or on a random held-out test set selected without ordering-based criteria?

  3. Material validation question: Can the authors compare CRA against equal-token or equal-cost baselines, including longer CPP prompting, generic self-reflection, structured career prompting, retrieval over the career history, and a simple profile-dependent authority/refusal controller?

  4. Evaluation validity question: Can the authors provide independent expert judgments of task correctness, supported actions, and decision authority, together with confidence intervals and paired statistical tests over tasks? Please also report the temperatures, seeds, number of repeated generations, complete evaluator prompts, and exact model/API snapshots used in the experiments.

8 comments
AuthorsOpenReview ↗

Author Response to Reviewer wgE9 (Weaknesses 1-1 and Questions 1)

We thank the reviewer for these comments.

W1-1: Profile ordering relies on task-specific auditing and relabeling. CRA also uses target occupation and ESCO metadata absent from the shared CPP prompt, raising concerns about information matching and the controller’s input interface.

Q1: Critical clarification: Does CRA receive the task’s target occupation, ESCO code, or any hidden benchmark metadata when selecting the behavioral regime? The shared CPP prompt does not contain this information, while Section B.3 says that the controller classifies the current occupation relative to the target occupation. If the metadata is supplied, can the authors rerun the comparison with identical information available to CPP? If it is inferred from the task, please provide the exact prompt and interface.

A1:

CRA receives task.target_esco_title and task.target_esco_code as structured task fields. The controller compares these fields with the persona’s current occupation and career records and uses occupational relevance and alignment between task requirements and career evidence to select a behavioral regime. It does not receive Senior/Mid/Junior/Unrelated slot labels, audited profile ranks, gold pairwise outcomes, or evaluator scores. We will make this input interface explicit in the revision.

We reran CPP on the same 100 tasks with the task metadata available to CRA. We inserted the following block immediately before [Workplace Scenario] in the shared CPP user prompt.

[Task Metadata]
Target occupation: {task.target_esco_title}
Target ESCO code: {task.target_esco_code}
Task statement: {task.task_statement}
Task domain: {task.task_domain}
Difficulty: {task.difficulty_proxy}
Required expertise: {task.expertise_level}
Risk domain: {task.risk_domain}
Expected output type: {task.expected_output_type}

Table 1. Ordering performance after providing CPP with the task metadata available to CRA. Each entry shows original CPP → metadata-matched CPP → CRA. All values are percentages.

BackboneOC ↑S-OC ↑B-OC ↑OVR ↓
Claude-Sonnet-573.7 → 74.6 → 82.365.0 → 66.3 → 80.182.5 → 82.8 → 84.523.9 → 22.9 → 8.8
Grok-4.672.2 → 74.8 → 78.761.3 → 64.0 → 80.383.0 → 85.7 → 77.027.7 → 21.0 → 9.7
DeepSeek-V3.270.7 → 68.9 → 80.760.7 → 54.0 → 81.780.7 → 83.9 → 79.730.7 → 31.6 → 9.1
DeepSeek-V4-Flash72.4 → 64.1 → 83.863.8 → 54.4 → 79.780.9 → 73.8 → 88.025.3 → 39.1 → 11.7
Gemini-3.7-Flash70.3 → 70.0 → 85.561.7 → 61.3 → 80.079.0 → 78.8 → 91.027.0 → 29.0 → 10.3
GPT-4.168.3 → 69.7 → 81.362.3 → 66.3 → 78.374.3 → 73.0 → 84.328.3 → 23.0 → 9.7
GPT-5.6-Sol68.3 → 65.8 → 83.858.0 → 55.3 → 80.778.7 → 76.3 → 87.029.3 → 35.3 → 9.3
GPT-5.6-Luna65.5 → 63.0 → 83.056.0 → 52.7 → 79.075.0 → 73.3 → 87.036.7 → 41.0 → 12.0
GPT-5.6-Terra63.8 → 64.2 → 82.557.3 → 54.7 → 79.770.3 → 73.7 → 85.335.0 → 38.3 → 10.0
GPT-5.564.0 → 63.3 → 84.057.7 → 53.4 → 76.770.3 → 73.7 → 91.322.7 → 35.8 → 14.0
Qwen3-32B57.0 → 64.2 → 85.052.3 → 58.3 → 78.361.7 → 70.0 → 91.739.3 → 34.3 → 13.3
Qwen3-8B61.8 → 64.0 → 80.857.0 → 57.3 → 78.066.7 → 70.7 → 83.734.3 → 35.0 → 10.7
Qwen3-14B56.8 → 59.7 → 79.253.7 → 52.0 → 75.760.0 → 67.3 → 82.745.7 → 40.3 → 12.7
Llama3.1-8B57.2 → 58.2 → 71.252.0 → 54.7 → 73.762.3 → 61.7 → 68.742.0 → 38.0 → 11.7
14-model macro-average65.9 → 66.0 → 81.658.5 → 57.5 → 78.773.2 → 74.6 → 84.432.0 → 33.2 → 10.9

Adding the metadata alone left CPP’s average OC nearly unchanged (65.9% → 66.0%). CRA outperformed metadata-matched CPP in both OC and OVR on all 14 models. Across models, OC increased from 66.0% to 81.6%, and OVR decreased from 33.2% to 10.9%. The average gains covered both ordering among career stages (S-OC, 57.5% → 78.7%) and comparisons involving Unrelated profiles (B-OC, 74.6% → 84.4%).

Table 2. Profile-aware answer quality under the same task-information comparison, macro-averaged across 14 models.

MeasureOriginal CPPMetadata-matched CPPCRA
Knowledge Fidelity ↑3.493.274.60
Behavior Coherence ↑3.863.404.63
Omission Rate ↓32.50%39.50%10.60%

CRA also retained higher Knowledge Fidelity and Behavior Coherence and a lower omission rate when CPP received the same task metadata. These results show that CRA’s advantage persists after matching access to the target occupation, ESCO code, and other task fields.

AuthorsOpenReview ↗

Author Response to Reviewer wgE9 (Weaknesses 1-2 and Questions 2)

W1-2: CRA is tested only on a fixed 100-task diagnostic subset selected using criteria that include ordering difficulty, a useful low-evidence boundary, and clear Senior support. No matched results are provided for the remaining 255 tasks or for an independently held-out random test set.

Q2: Material validation question: How were the 100 CapRoleBench-100 tasks selected, and were the selection criteria fixed before observing CPP or CRA outputs? What are CRA’s results on the remaining 255 tasks, or on a random held-out test set selected without ordering-based criteria?

A2: CapRoleBench-100 was constructed as a diagnostic stress test. We retained medium and hard tasks requiring intermediate or expert-level work, with adequate Senior performance under CPP. Selection considered direct-ordering difficulty, task complexity, a meaningful capability boundary for lower-evidence profiles, and clear support from the Senior profile’s career evidence. CPP outputs informed this selection. The selection criteria and the 100 task IDs were fixed before CRA evaluation, and no CRA responses or scores were used. The resulting subset contains 62 medium and 38 hard tasks.

We have now evaluated CRA on all 255 remaining tasks across seven backbones spanning six model families, without further filtering based on ordering performance. These tasks comprise 96 easy, 101 medium, and 58 hard cases. Together with the original diagnostic evaluation, this covers all 355 benchmark tasks for these seven backbones. Table 3 compares CPP and CRA on the same available task–profile records within each model.

Table 3. CPP → CRA results on all 255 remaining benchmark tasks.

BackboneOC ↑S-OC ↑B-OC ↑OVR ↓KF ↑BC ↑OMR ↓
Claude-Sonnet-569.3 → 74.662.7 → 71.275.8 → 77.917.4 → 10.64.25 → 4.664.70 → 4.8711.2 → 5.6
DeepSeek-V4-Flash73.1 → 81.964.7 → 77.581.4 → 86.423.5 → 7.04.04 → 4.904.66 → 4.9810.4 → 2.7
Gemini-3.7-Flash69.9 → 78.063.5 → 76.376.3 → 79.628.0 → 11.93.91 → 4.604.54 → 4.8715.2 → 6.0
GPT-5.6-Sol67.1 → 74.660.5 → 73.973.6 → 75.428.1 → 11.24.03 → 4.784.63 → 4.9111.8 → 5.0
Llama3.1-8B64.1 → 68.661.3 → 68.466.9 → 68.731.6 → 16.93.16 → 3.573.86 → 3.8843.6 → 41.1
Qwen3-32B68.4 → 80.762.5 → 78.774.2 → 82.628.2 → 9.43.49 → 4.184.27 → 4.7225.8 → 15.7
Qwen3-8B64.6 → 78.558.8 → 78.470.5 → 78.633.5 → 10.23.37 → 4.044.05 → 4.5733.8 → 20.3
Model macro-average68.1 → 76.762.0 → 74.974.1 → 78.527.2 → 11.03.75 → 4.394.39 → 4.6921.7 → 13.8

All seven backbones achieved higher OC, lower OVR, and fewer omissions with CRA. Across models, OC increased from 68.1% to 76.7%, and OVR decreased from 27.2% to 11.0%. Both seniority ordering and comparisons involving Unrelated profiles improved. The profile-aware measures also improved on average, with KF rising from 3.75 to 4.39, BC from 4.39 to 4.69, and OMR falling from 21.7% to 13.8%.

These results extend the evidence across the entire set of tasks outside CapRoleBench-100, including every easy task. CRA’s improvements in ordering and profile-aware answer quality therefore extend beyond the diagnostic tasks selected using CPP performance.

AuthorsOpenReview ↗

Author Response to Reviewer wgE9 (Weaknesses 2-1 and Questions 3)

W2-1: CRA uses approximately 7.05 times the tokens and 4.90 times the cost of CPP, multiple LLM-generated intermediate states, and up to two self-revisions, but there is no equal-budget baseline using generic self-critique, longer prompting, structured career prompts, retrieval, or explicit refusal/authority control.

Q3: Material validation question: Can the authors compare CRA against equal-token or equal-cost baselines, including longer CPP prompting, generic self-reflection, structured career prompting, retrieval over the career history, and a simple profile-dependent authority/refusal controller?

A3: We added two baselines that incorporate these strategies and configured them using CRA’s token consumption as the budget target for each backbone. The evaluation covers seven backbones from six model families. Across these models, CRA achieved higher average Ordering Consistency (OC), lower Overreach Rate (OVR), and better profile-aware answer quality than both baselines.

  • Direct evidence-and-scope prompting (DESP) extends CPP with explicit instructions to use career evidence, identify supported work, and stay within the role’s authority. It uses the additional budget to generate independent candidate answers and select the best one.
  • The simpler contract-and-revision workflow (SCR) retrieves relevant career records, writes a compact contract specifying supported actions and authority boundaries, and applies generic critique and revision. Its critique checks for unsupported authority, unnecessary refusal, and omitted work the profile could perform.

These workflows incorporate the reviewer’s suggested longer prompting, structured use of career evidence, retrieval, profile-dependent scope control, and self-reflection. We calibrated candidate counts and revision limits on separate examples using token consumption, without consulting evaluation scores. We used Qwen3-32B as the evaluator for all three methods. Table 4 reports actual mean input-plus-output tokens across every inference call. Within each backbone, results use the same task–profile items completed by all three methods.

Table 4. Actual token consumption and ordering performance for the two simpler workflows and CRA. Values within each cell are ordered DESP / SCR / CRA. Token counts are per task–profile response.

BackboneTokens per task–profile responseOC ↑ (%)OVR ↓ (%)
Llama3.1-8B19,614 / 17,464 / 18,76962.2 / 70.2 / 68.534.3 / 32.0 / 17.7
Qwen3-8B13,677 / 11,572 / 12,68370.3 / 66.3 / 77.326.7 / 30.7 / 14.3
Qwen3-32B14,277 / 14,167 / 14,74666.2 / 70.8 / 83.035.7 / 28.3 / 11.3
Claude-Sonnet-521,640 / 7,981 / 24,45068.4 / 69.9 / 72.433.0 / 25.9 / 13.1
DeepSeek-V4-Flash12,795 / 9,292 / 14,48255.6 / 65.4 / 81.849.3 / 30.7 / 11.3
Gemini-3.7-Flash22,096 / 8,612 / 21,73865.7 / 69.0 / 81.327.0 / 26.0 / 12.3
GPT-5.6-Sol39,973 / 23,476 / 19,57957.2 / 62.0 / 77.742.3 / 31.7 / 12.7

In nine of the 14 baseline comparisons, actual token use was within 12% of CRA’s. CRA achieved lower OVR in all nine and higher OC in eight. The exception was SCR on Llama3.1-8B, which achieved higher OC (70.2% vs. 68.5%) but also substantially higher OVR (32.0% vs. 17.7%). Across all seven backbones, CRA had lower OVR than both baselines, with paired task-bootstrap 95% confidence intervals excluding zero for all 14 differences. Its OC exceeded DESP’s on all seven backbones and SCR’s on six.

Table 5. Ordering performance and profile-aware answer quality, macro-averaged across the seven backbones.

MethodOC ↑ (%)S-OC ↑ (%)B-OC ↑ (%)OVR ↓ (%)KF ↑BC ↑Omission rate ↓ (%)
DESP63.657.469.935.53.7184.37722.3
SCR67.760.175.329.34.0354.56817.5
CRA77.473.781.313.24.3874.71914.8

Across backbones, CRA improved OC by 13.8 percentage points over DESP and 9.7 points over SCR, while reducing OVR by 22.3 and 16.1 points, respectively. CRA also achieved higher Knowledge Fidelity and Behavior Coherence and fewer omissions than either baseline. These results support the value of CRA’s full workflow when compared with simpler methods that use additional inference to select answers or combine career retrieval, authority guidance, and critique and revision.

Reviewer_Qjf5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CapRoleBench, with 355 workplace tasks spanning 355 occupations, each paired with Senior, Mid, Junior, and Unrelated career profiles. It evaluates whether responses differ appropriately in expertise and decision authority. The proposed CapRoleAgent (CRA) extracts a capability graph, aligns career evidence with task requirements, selects a behavioral contract, generates a response, and checks it for overreach and omissions. All model-based stages use the same frozen backbone. Direct career-profile prompting is evaluated across 14 models on the full benchmark. Matched CRA comparisons use a selected 100-task subset, where macro-average Ordering Consistency rises from 65.9% to 81.6% and Overreach Rate falls from 32.0% to 10.9%. Component ablations, three LLM evaluators, and a persona-blind human ranking study on 25 tasks provide additional checks.

Critical strengths

1.The paper addresses an overlooked aspect of role fidelity by varying career evidence while keeping tasks fixed. Its distinction between technical expertise and decision authority is useful for professional role simulation.

2.CRA improves capability ordering across 14 models, with supporting results from multiple LLM judges and human evaluation. The ablations help identify which components drive these gains.

Critical weaknesses

1.Correct capability ordering does not establish that each answer fits its role or remains useful. Profile-aware expert evaluation should assess these separately, especially given the worse omission and coherence scores for some smaller models.

2.CRA uses about seven times as many tokens as direct prompting, but lacks stronger baselines. Comparisons with simpler scope-setting prompts and critique-and-revision methods are needed to justify the full framework.

3.The 100-task subset was selected partly for difficulty under direct prompting, which may favor CRA. Results on a random subset or the full benchmark, together with independent validation of profile ordering, would strengthen the conclusions.

Questions for authors
  1. Can you separately evaluate task utility and unsupported expertise or authority using profile-aware domain experts? How often are ties appropriate across career profiles?

  2. How does CRA compare with direct evidence-and-scope prompting or a simpler contract-and-revision workflow at a similar token budget?

  3. Were the 100 diagnostic tasks selected before inspecting CRA results? Do the gains persist on a randomly sampled subset, including the weaker models' omission and coherence scores?

  4. Who audited the Senior/Mid/Junior ordering, what criteria did they use, and what agreement was obtained? How sensitive are results to the five-year threshold for Complete core and the rules based on current occupation?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer Qjf5 (Weaknesses 1and Questions 1)

We thank the reviewer for these comments.

W1: Correct capability ordering does not establish that each answer fits its role or remains useful. Profile-aware expert evaluation should assess these separately, especially given the worse omission and coherence scores for some smaller models.

Q1: Can you separately evaluate task utility and unsupported expertise or authority using profile-aware domain experts? How often are ties appropriate across career profiles?

A1: We added two human evaluations on the original 25 tasks. The first assessed usefulness separately from unsupported expertise and unauthorized decisions. The second used only tasks and career profiles to judge whether two profiles could reasonably give equally good answers.

1. Task utility and capability boundaries

Five professors and five PhD students researching role-playing LLMs took part. They used the complete profile, including seniority, to judge what the role could do. CPP and CRA answers for the same task, model, and profile appeared in randomized A/B order, with method and model identities hidden. They rated each answer separately for task utility, correctness for the role, unsupported expertise, unauthorized decisions. The study covered four profile levels and two models (GPT-5.6-Sol and Qwen3-32B). Three evaluators assessed each matched pair, yielding 600 answer ratings per method.

Table 1. Profile-aware human ratings of task utility, correctness, and capability-boundary violations. Values are percentages of ratings, shown as CPP → CRA. Each profile has 150 ratings per method.

ProfileFully useful ↑No practical help ↓Correct for the role ↑Unsupported expertise ↓Unauthorized decision ↓
Senior77.3 → 84.70.7 → 1.374.0 → 81.36.0 → 2.06.7 → 3.3
Mid-level60.7 → 73.34.7 → 1.360.7 → 81.319.3 → 5.313.3 → 4.0
Junior48.7 → 56.010.0 → 6.056.0 → 70.026.0 → 4.726.7 → 4.0
Unrelated41.3 → 40.046.0 → 25.346.7 → 58.041.3 → 7.336.7 → 3.3
Overall57.0 → 63.515.3 → 8.559.3 → 72.723.2 → 4.820.8 → 3.7

Evaluators rated CRA answers as fully useful more often overall (63.5% vs. 57.0% for CPP) and as correct for the role more often at every profile level. They also flagged fewer unsupported claims of expertise (139/600 → 29/600) and fewer unauthorized decisions (125/600 → 22/600). The reductions were 18.3 percentage points for unsupported expertise (task-clustered 95% CI, 12.2–25.0) and 17.2 points for unauthorized decisions (95% CI, 12.3–22.5).

The Junior and Unrelated results directly address whether answers remain useful and fit the role when the profile provides less evidence of expertise. Evaluators saw the full career profile and rated each answer against that profile, independently of its rank. For Junior profiles, “fully useful” ratings rose from 48.7% to 56.0% and “correct for the role” ratings rose from 56.0% to 70.0%, while unsupported expertise fell from 26.0% to 4.7%. For Unrelated profiles, “fully useful” ratings were similar under CPP and CRA (41.3% and 40.0%), but “no practical help” ratings fell from 46.0% to 25.3%. More answers were judged correct for the role (46.7% → 58.0%), and fewer were flagged for unauthorized decisions (36.7% → 3.3%).

For the two 8B models, the original 100 tasks showed higher omission rates and lower Behavior Coherence with CRA. On all 255 remaining tasks, omissions decreased for both. Behavior Coherence improved for Qwen3-8B (4.05 → 4.57) and remained nearly unchanged for Llama3.1-8B (3.86 → 3.88).

2. When can different career profiles reasonably tie?

For each of the 25 tasks, three evaluators saw the task and all four career profiles before seeing any generated answers for this assessment. For each of the six profile pairs, they judged whether both roles could give answers that were equally correct, useful, and appropriate for their respective profiles. We counted a potential tie when at least two of the three evaluators agreed.

Table 2. Task–profile pairs for which evaluators judged that equally good, role-appropriate answers were possible.

Career-profile pairPotentially appropriate tie
Senior–Mid-level5/25 (20%)
Mid-level–Junior3/25 (12%)
Senior–Junior1/25 (4%)
Junior–Unrelated1/25 (4%)
Mid-level–Unrelated0/25 (0%)
Senior–Unrelated0/25 (0%)

Evaluators judged a tie potentially appropriate in only 10 of the 150 comparisons (6.7%). Eight of those ten involved adjacent career stages, and only one of the 75 comparisons involving an Unrelated profile allowed for a reasonable tie. These judgments support the use of strict ordering for most comparisons in our evaluation, particularly when the profiles differ substantially in their evidence of relevant expertise.

AuthorsOpenReview ↗

Author Response to Reviewer Qjf5 (Weaknesses 2 and Questions 2)

W2: CRA uses about seven times as many tokens as direct prompting, but lacks stronger baselines. Comparisons with simpler scope-setting prompts and critique-and-revision methods are needed to justify the full framework.

Q2: How does CRA compare with direct evidence-and-scope prompting or a simpler contract-and-revision workflow at a similar token budget?

A2: We added both baselines suggested by the reviewer to test whether CRA retains its advantage under comparable token budgets. For each backbone, we used CRA’s token consumption as the budget target when configuring the baselines. Across seven backbones, CRA achieved higher average Ordering Consistency (OC) and lower average Overreach Rate (OVR) than both alternatives.

Direct evidence-and-scope prompting (DESP): instructs the model to use the career evidence and stay within the role’s scope. It uses the additional budget to generate independent answers and select the best one.

The simpler contract-and-revision workflow (SCR): retrieves relevant career evidence, writes a short contract specifying what the role can do, and uses generic critique and revision to refine the answer.

All three methods used the same tasks, career profiles, backbones, and evaluation rubric. We used Qwen3-32B as the evaluator for all three methods. Table 3 reports the actual input and output tokens across all inference calls. Within each backbone, results cover the task–profile pairs completed by all three methods.

Table 3. Actual token use and calibration performance by backbone. Values in each cell are ordered DESP / SCR / CRA. Token counts are per task–profile pair.

BackboneTokens per task–profile pairOC ↑ (%)OVR ↓ (%)
Llama3.1-8B19,614 / 17,464 / 18,76962.2 / 70.2 / 68.534.3 / 32.0 / 17.7
Qwen3-8B13,677 / 11,572 / 12,68370.3 / 66.3 / 77.326.7 / 30.7 / 14.3
Qwen3-32B14,277 / 14,167 / 14,74666.2 / 70.8 / 83.035.7 / 28.3 / 11.3
Claude-Sonnet-521,640 / 7,981 / 24,45068.4 / 69.9 / 72.433.0 / 25.9 / 13.1
DeepSeek-V4-Flash12,795 / 9,292 / 14,48255.6 / 65.4 / 81.849.3 / 30.7 / 11.3
Gemini-3.7-Flash22,096 / 8,612 / 21,73865.7 / 69.0 / 81.327.0 / 26.0 / 12.3
GPT-5.6-Sol39,973 / 23,476 / 19,57957.2 / 62.0 / 77.742.3 / 31.7 / 12.7

In nine of the 14 baseline comparisons, actual token use was within 12% of CRA’s. CRA achieved lower OVR in all nine and higher OC in eight. The one exception was SCR on Llama3.1-8B, where SCR had slightly higher OC (70.2% vs. 68.5%) but substantially higher OVR (32.0% vs. 17.7%). Across all seven backbones, CRA had lower OVR than both baselines on every model, with paired task-bootstrap 95% confidence intervals excluding zero for all 14 differences.

Table 4. Cross-profile ordering and profile-aware answer quality, macro-averaged across the seven backbones.

MethodOC ↑ (%)S-OC ↑ (%)B-OC ↑ (%)OVR ↓ (%)KF ↑BC ↑OMR ↓ (%)
DESP63.657.469.935.53.7184.37722.3
SCR67.760.175.329.34.0354.56817.5
CRA77.473.781.313.24.3874.71914.8

Across backbones, CRA improved OC by 13.8 percentage points over DESP and 9.7 points over SCR, while reducing OVR by 22.3 and 16.1 points, respectively. The benefit also extended to the answers themselves. CRA achieved higher knowledge fidelity and behavior coherence than either baseline, and fewer answers omitted work the role could perform (14.8% for CRA, 17.5% for SCR, and 22.3% for DESP).

AuthorsOpenReview ↗

Author Response to Reviewer Qjf5 (Weaknesses 3 and Questions 3)

W3: The 100-task subset was selected partly for difficulty under direct prompting, which may favor CRA. Results on a random subset or the full benchmark, together with independent validation of profile ordering, would strengthen the conclusions.

Q3: Were the 100 diagnostic tasks selected before inspecting CRA results? Do the gains persist on a randomly sampled subset, including the weaker models' omission and coherence scores?

A3: Yes. CapRoleBench-100 was fixed before we inspected any CRA responses or scores. We selected the diagnostic tasks using task attributes and CPP-only evidence, including direct-ordering difficulty, task complexity, a meaningful capability boundary for lower-evidence profiles, and clear support for the Senior profile. CRA results played no part in selecting the tasks.

To test whether the gains extended beyond this selection, we evaluated CPP and CRA on all 255 remaining tasks across seven backbones. These tasks comprise 96 easy, 101 medium, and 58 hard cases. Table 5 compares the methods using the same available task–profile records within each backbone.

Table 5. CPP → CRA results on all 255 remaining benchmark tasks.

BackboneOC ↑S-OC ↑B-OC ↑OVR ↓KF ↑BC ↑OMR ↓
Claude-Sonnet-569.3 → 74.662.7 → 71.275.8 → 77.917.4 → 10.64.25 → 4.664.70 → 4.8711.2 → 5.6
DeepSeek-V4-Flash73.1 → 81.964.7 → 77.581.4 → 86.423.5 → 7.04.04 → 4.904.66 → 4.9810.4 → 2.7
Gemini-3.7-Flash69.9 → 78.063.5 → 76.376.3 → 79.628.0 → 11.93.91 → 4.604.54 → 4.8715.2 → 6.0
GPT-5.6-Sol67.1 → 74.660.5 → 73.973.6 → 75.428.1 → 11.24.03 → 4.784.63 → 4.9111.8 → 5.0
Llama3.1-8B64.1 → 68.661.3 → 68.466.9 → 68.731.6 → 16.93.16 → 3.573.86 → 3.8843.6 → 41.1
Qwen3-32B68.4 → 80.762.5 → 78.774.2 → 82.628.2 → 9.43.49 → 4.184.27 → 4.7225.8 → 15.7
Qwen3-8B64.6 → 78.558.8 → 78.470.5 → 78.633.5 → 10.23.37 → 4.044.05 → 4.5733.8 → 20.3
Model macro-average68.1 → 76.762.0 → 74.974.1 → 78.527.2 → 11.03.75 → 4.394.39 → 4.6921.7 → 13.8

CRA achieved higher OC and lower OVR on every backbone. Across the seven models, average OC increased by 8.6 percentage points, OVR decreased by 16.2 points, and OMR decreased by 7.9 points. Knowledge Fidelity and Behavior Coherence also increased on average. These results show that the gains extend across the remaining benchmark tasks, covering a broader range of difficulty than the original diagnostic subset.

On the remaining 255 tasks, both 8B models showed fewer omissions and stable or improved Behavior Coherence with CRA. Qwen3-8B’s OMR fell from 33.8% to 20.3% and its BC rose from 4.05 to 4.57. Llama3.1-8B’s OMR fell from 43.6% to 41.1%, while BC remained nearly unchanged (3.86 → 3.88). These measures remain challenging for the two smaller models, as observed on the original 100 tasks, but the remaining-task results show improvement rather than further deterioration.

The independent validation of profile ordering is reported in our response to Q1, where evaluators assessed the tasks and career profiles without seeing generated answers.

Reviewer_k92yOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies capability calibration in professional role-playing: whether language models restrict their expertise, response scope, and decision authority to what a supplied career history supports. It introduces CapRoleBench, comprising 355 workplace tasks paired with four career profiles each, yielding 1,420 instances. Tasks remain fixed across profiles, allowing the benchmark to examine how responses change with task-specific career evidence.

The paper also proposes CapRoleAgent, a training-free framework that builds a capability graph from career history, matches task requirements to supporting evidence, sets a response contract, generates an answer, and checks it for unsupported claims or omitted supported work. Its intended behavior is to complete justified work while qualifying or escalating requests beyond the persona’s capabilities.

Experiments evaluate 14 language models, with the main method comparison conducted on a selected 100-task subset. Relative to direct career-profile prompting, average ordering consistency increases from 65.9% to 81.6%, while the reported overreach rate decreases from 32.0% to 10.9%. Additional judges, human rankings, and component ablations support the reported improvements, with monitoring and behavioral control contributing most. Overall, the contribution combines a career-evidence benchmark with a mechanism for controlling response scope, although the headline metrics primarily assess response ordering rather than directly measuring unsupported expertise.

Critical strengths

1 - The benchmark provides a carefully controlled setting for studying how career evidence affects role-playing. 2 - The improvements are consistent across models and supported by several complementary evaluations. CapRoleAgent improves ordering consistency and reduces the reported overreach rate across all 14 evaluated models on the matched subset. The authors further examine robustness using three judges on seven models and a human evaluation covering 25 tasks and two models.

Critical weaknesses

The main metrics do not directly establish that responses respect the persona’s capability boundaries. My main concern is the definition of Overreach Rate. A case counts as overreach when a lower-evidence persona’s answer is judged better than the Senior persona’s answer. However, this can happen because the Senior answer is weak, even if the other answer stays entirely within its supported capabilities. Conversely, multiple personas could make unsupported claims while preserving the expected ranking. Ordering Consistency also requires a strict ordering and counts ties as incorrect, although two profiles may legitimately provide equally good, fully supported answers. (Appendix C) Consequently, these scores can improve when lower-evidence answers become less useful, without demonstrating a reduction in unsupported expertise or unauthorized decisions. The profile-aware evaluations partly address this concern, but the human study evaluates the same ordering assumption: raters do not see the profiles and do not directly compare CPP and CRA answers for the same profile.

Questions for authors

Does CapRoleAgent preserve usefulness for each profile? Could you provide blinded CPP-versus-CRA comparisons for the same task and profile, and explain the increased omissions and reduced coherence for Llama3.1-8B and Qwen3-8B?

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer k92y (Weaknesses 1and Questions 1-1)

We thank the reviewer for these comments.

W1: The ordering metrics do not directly establish whether individual answers respect capability boundaries or remain useful. OVR may reflect a weak Senior answer or miss unsupported claims when the expected ranking holds. OC also penalizes legitimate ties. These metrics could therefore improve when lower-evidence answers become less useful. The existing human study hides the profiles and does not directly compare CPP and CRA for the same task and profile.

Q1-1: Does CapRoleAgent preserve usefulness for each profile? Could you provide blinded CPP-versus-CRA comparisons for the same task and profile?

A1: We conducted this blinded evaluation on the 25 tasks from our original human study. CRA received more “useful” ratings for Senior, Mid-level, and Junior profiles. For Unrelated profiles, the two methods received similar shares of “useful” ratings, and CRA received substantially fewer “no practical help” ratings. Evaluators also flagged fewer instances of unsupported expertise and unauthorized decisions with CRA at every profile level.

We compared CPP and CRA on the 25 tasks from our original human study. Each pair contained two answers generated by the same model (GPT-5.6-Sol or Qwen3-32B) for the same task and career profile. Ten evaluators took part: five professors and five PhD students, all of whom conduct research on role-playing LLMs. Three evaluators were assigned to each task. They saw the full career profile, including seniority, and the two answers in randomized A/B order. Both method and model identities were hidden. Evaluators first rated each answer separately for usefulness, correctness for the role, unsupported expertise, unauthorized decisions, and critical omissions. They then chose which answer better fit the task and profile, with the option to select a tie. Across 25 tasks, four profiles, and two models, this yielded 200 matched answer pairs, 600 preference judgments, and 1,200 individual answer ratings.

Table 1. Human ratings of usefulness, correctness, and capability-boundary violations for the same task and profile. Values are percentages of ratings, shown as CPP → CRA. Each profile has 150 ratings per method.

ProfileFully useful ↑No practical help ↓Correct for the role ↑Unsupported expertise ↓Unauthorized decision ↓
Senior77.3 → 84.70.7 → 1.374.0 → 81.36.0 → 2.06.7 → 3.3
Mid-level60.7 → 73.34.7 → 1.360.7 → 81.319.3 → 5.313.3 → 4.0
Junior48.7 → 56.010.0 → 6.056.0 → 70.026.0 → 4.726.7 → 4.0
Unrelated41.3 → 40.046.0 → 25.346.7 → 58.041.3 → 7.336.7 → 3.3
Overall57.0 → 63.515.3 → 8.559.3 → 72.723.2 → 4.820.8 → 3.7

The Junior and Unrelated results speak directly to the concern that stronger ordering could come from making lower-evidence answers less useful. For Junior profiles, “fully useful” ratings rose from 48.7% to 56.0%, while unsupported expertise fell from 26.0% to 4.7%. For Unrelated profiles, “fully useful” ratings were close (41.3% for CPP and 40.0% for CRA), while “no practical help” fell from 46.0% to 25.3% and unauthorized decisions fell from 36.7% to 3.3%. Evaluators saw the full career profile and assessed each answer against the capabilities described in that profile. A weak Senior answer therefore could not cause another profile’s answer to be flagged, and evaluators could still flag unsupported claims in answers from several profiles even if those answers followed the expected ranking.

Table 2. Blinded CPP–CRA preferences for the same task, model, and profile. Net preference is the percentage-point difference between CRA and CPP preferences across all judgments, including ties. The 95% confidence intervals were obtained by resampling whole tasks.

ProfileCRA preferredCPP preferredTieNet CRA preference, pp (95% CI)
Senior772944+32.0 (+14.7, +48.7)
Mid-level743937+23.3 (+6.0, +40.7)
Junior754827+18.0 (+2.0, +33.3)
Unrelated894516+29.3 (+14.7, +44.7)
Overall315161124+25.7 (+16.2, +35.2)

We will clarify that OC measures strict cross-profile ordering and OVR counts lower-evidence answers outranking Senior answers. Ties remain in OC’s denominator without credit for correct ordering. The human study complements these metrics and our existing profile-aware evaluations. Evaluators saw full profiles, rated answers, and compared CPP and CRA. They recorded 124 ties in 600 judgments and preferred CRA more often at every level. Ratings show gains beyond ordering, with fewer unsupported claims and unauthorized decisions at all four levels and more fully useful answers overall (63.5% vs. 57.0% for CPP).

AuthorsOpenReview ↗

Author Response to Reviewer k92y (Questions 1-2)

Q1-2: Explain the increased omissions and reduced coherence for Llama3.1-8B and Qwen3-8B?

A2: Our analysis suggests that the difficulty for the two 8B models was carrying a judgment about the role’s capabilities through several stages of a demanding task. With CPP, the model answers directly from the career profile. With CRA, the same model decides which actions the profile supports, records them in a behavioral contract, drafts the answer, and checks its own work. When a task combines a decision that requires a specialist with an action the role can take, the model needs to preserve that distinction at each stage.

The original 100-task subset was selected in part for task complexity and for cases with a meaningful capability boundary. It contains 62 medium and 38 hard tasks. On this subset, both 8B models acknowledged capability or authority limits more often under CRA than under CPP (22.5% → 88.2% for Llama3.1-8B and 28.2% → 67.0% for Qwen3-8B). Knowledge Fidelity also improved for both. To understand why they nevertheless omitted more work, three PhD students researching role-playing LLMs compared CPP and CRA answers for the same tasks and profiles and examined CRA’s intermediate execution traces. They identified difficulties at three stages.

  1. Interpreting the task. In a technician-dispatch task, Qwen3-8B treated HVAC and security diagnostics as work the dispatcher had to perform personally. The profile supported assigning qualified technicians to handle those issues. Qwen therefore wrote a contract that restricted the dispatcher too broadly. Its final answer made only part of the assignment before recommending escalation.
  2. Executing the contract. In a contract-review task, Llama3.1-8B correctly specified that the Senior profile could recommend a limited change to a meeting. When drafting the answer, it stopped at suggesting further review. The contract contained the right judgment, but the model left the permitted recommendation out of its response.
  3. Reviewing the answer. In CRA’s fifth stage (Feedback and Adjustment), the same 8B model acted as the reviewer of its own draft. In both cases, it missed an action the role could still take, so the omission remained in the final answer.

Both 8B models followed the expected cross-profile ordering more often with CRA. OC rose from 57.2% to 71.2% for Llama3.1-8B and from 61.8% to 80.8% for Qwen3-8B. Lower-evidence answers also outranked Senior answers less often, reducing OVR from 42.0% to 11.7% and from 34.3% to 10.7%, respectively. A Senior answer can still rank above a Junior answer even if it leaves out a recommendation the Senior profile is qualified to make. The missing recommendation counts toward OMR, and ending with a general call for review can lower BC when the task calls for a concrete next step. On the same 100 tasks, all 12 other backbones, including Qwen3-14B and Qwen3-32B, achieved both lower OMR and higher BC with CRA (Table 5).

We also evaluated both 8B models on all 255 remaining tasks. This set covers a broader range of difficulty and includes 96 easy tasks.

Table 3. Behavior Coherence (BC, 1–5) and Omission Rate (OMR) for the two 8B models on the original 100 tasks and the remaining 255 tasks.

ModelTask subsetBC ↑ (CPP → CRA)OMR ↓ (CPP → CRA)
Llama3.1-8BOriginal 1004.19 → 3.5836.2% → 42.2%
Llama3.1-8BRemaining 2553.86 → 3.8843.6% → 41.1%
Qwen3-8BOriginal 1004.38 → 4.0427.0% → 30.2%
Qwen3-8BRemaining 2554.05 → 4.5733.8% → 20.3%

On the remaining 255 tasks, CRA reduced omissions for both models. Qwen3-8B’s OMR fell by 13.5 percentage points and its BC rose by 0.52 points. Llama3.1-8B’s OMR fell by 2.5 percentage points while its BC remained nearly unchanged. This difference across subsets is consistent with the explanation from the traces. On demanding tasks, an 8B model can set the wrong boundary early or fail to act on a correct one. If its own review misses the error, the extra steps can leave the answer less complete than direct prompting. Across the broader task mix, both models were able to use the same procedure with fewer omissions.

AuthorsOpenReview ↗

Gentle Reminder: Rebuttal Phase Ending Soon

Dear Reviewer k92y,

I hope this message finds you well. We recently submitted our author response and would like to kindly remind you to review it. Given the approaching deadline, we would appreciate it if you could confirm whether our response has addressed your concerns. We also value any additional suggestions you may have.

Thank you for your attention and support.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.