- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies career-grounded capability calibration in role-playing language models. The goal is for a model to express expertise, execution scope, and decision authority supported by a persona’s career history, rather than using its general capabilities to answer every task as an expert. The authors introduce CapRoleBench, which fixes 355 workplace tasks and varies four career profiles—Senior, Mid, Junior, and Unrelated—yielding 1,420 controlled task–profile instances. They also propose CapRoleAgent (CRA), a training-free inference-time framework that constructs a capability graph from the career history, monitors task requirements and evidence alignment, selects a behavioral regime, generates a response, and performs feedback-based revision.
CPP and CRA are compared on a fixed 100-task diagnostic subset across 14 language models. CRA improves macro-average Ordering Consistency from 65.9% to 81.6% and reduces Overreach Rate from 32.0% to 10.9%. The paper also reports profile-aware LLM-judge scores, leave-one-component-out ablations, results from three evaluators, and a small persona-blind human ranking study. The reported improvements are strongest for the Monitoring and Control components.
Critical strengths
-
Controlled benchmark framing. CapRoleBench fixes the workplace task while varying only the career evidence, which isolates a meaningful distinction between solving a task and expressing capability that is appropriate for a persona. This is a useful problem formulation for role-playing and agent evaluation.
-
Broad empirical coverage. The paper evaluates 14 language models, reports multiple automated evaluators, component ablations, and a persona-blind human ranking study. The improvement in ordering metrics is directionally consistent across the evaluated models and evaluators, which makes the phenomenon worth further investigation even though the current validation has important limitations.
Critical weaknesses
-
The central CRA evaluation is vulnerable to benchmark-selection and information-leakage effects. CRA is tested only on a fixed 100-task diagnostic subset selected using criteria that include ordering difficulty, a useful low-evidence boundary, and clear Senior support. No matched results are provided for the remaining 255 tasks or for an independently held-out random test set. Moreover, the Senior/Mid/Junior ordering is created through task-specific auditing and relabeling of slices from the same career trajectory, while Unrelated is defined using a different occupation group. The CRA controller is then described as using the current occupation’s relation to the target occupation and ESCO code, although that target metadata is absent from the shared CPP prompt. If this metadata is supplied to CRA, the comparison is not information matched; if it is inferred from the task, the exact interface is unclear. The observed OC and OVR gains may therefore reflect matching the benchmark’s construction rules rather than general capability calibration.
-
The causal attribution and practical utility of CRA are underdetermined. CRA uses approximately 7.05 times the tokens and 4.90 times the cost of CPP, multiple LLM-generated intermediate states, and up to two self-revisions, but there is no equal-budget baseline using generic self-critique, longer prompting, structured career prompts, retrieval, or explicit refusal/authority control. The same general type of LLM judges the profile-blind ordering and profile-aware metrics, and GPT-4.1 is both the primary evaluator and one of the evaluated backbones. No objective expert labels for task correctness, supported authority, or practical helpfulness are reported, and no confidence intervals, paired statistical tests, or repeated generations are provided. The large increase in Refusal Rate is not quality-sensitive, and Behavior Coherence or Omission Rate worsens for Llama3.1-8B and Qwen3-8B. Thus higher ordering consistency may partly be achieved by more cautious or refusal-heavy behavior rather than by better useful assistance within the supported capability boundary.
Questions for authors
-
Critical clarification: Does CRA receive the task’s target occupation, ESCO code, or any hidden benchmark metadata when selecting the behavioral regime? The shared CPP prompt does not contain this information, while Section B.3 says that the controller classifies the current occupation relative to the target occupation. If the metadata is supplied, can the authors rerun the comparison with identical information available to CPP? If it is inferred from the task, please provide the exact prompt and interface.
-
Material validation question: How were the 100 CapRoleBench-100 tasks selected, and were the selection criteria fixed before observing CPP or CRA outputs? What are CRA’s results on the remaining 255 tasks, or on a random held-out test set selected without ordering-based criteria?
-
Material validation question: Can the authors compare CRA against equal-token or equal-cost baselines, including longer CPP prompting, generic self-reflection, structured career prompting, retrieval over the career history, and a simple profile-dependent authority/refusal controller?
-
Evaluation validity question: Can the authors provide independent expert judgments of task correctness, supported actions, and decision authority, together with confidence intervals and paired statistical tests over tasks? Please also report the temperatures, seeds, number of repeated generations, complete evaluator prompts, and exact model/API snapshots used in the experiments.
8 comments
Author Response to Reviewer wgE9 (Weaknesses 1-2 and Questions 2)
W1-2: CRA is tested only on a fixed 100-task diagnostic subset selected using criteria that include ordering difficulty, a useful low-evidence boundary, and clear Senior support. No matched results are provided for the remaining 255 tasks or for an independently held-out random test set.
Q2: Material validation question: How were the 100 CapRoleBench-100 tasks selected, and were the selection criteria fixed before observing CPP or CRA outputs? What are CRA’s results on the remaining 255 tasks, or on a random held-out test set selected without ordering-based criteria?
A2: CapRoleBench-100 was constructed as a diagnostic stress test. We retained medium and hard tasks requiring intermediate or expert-level work, with adequate Senior performance under CPP. Selection considered direct-ordering difficulty, task complexity, a meaningful capability boundary for lower-evidence profiles, and clear support from the Senior profile’s career evidence. CPP outputs informed this selection. The selection criteria and the 100 task IDs were fixed before CRA evaluation, and no CRA responses or scores were used. The resulting subset contains 62 medium and 38 hard tasks.
We have now evaluated CRA on all 255 remaining tasks across seven backbones spanning six model families, without further filtering based on ordering performance. These tasks comprise 96 easy, 101 medium, and 58 hard cases. Together with the original diagnostic evaluation, this covers all 355 benchmark tasks for these seven backbones. Table 3 compares CPP and CRA on the same available task–profile records within each model.
Table 3. CPP → CRA results on all 255 remaining benchmark tasks.
| Backbone | OC ↑ | S-OC ↑ | B-OC ↑ | OVR ↓ | KF ↑ | BC ↑ | OMR ↓ |
|---|---|---|---|---|---|---|---|
| Claude-Sonnet-5 | 69.3 → 74.6 | 62.7 → 71.2 | 75.8 → 77.9 | 17.4 → 10.6 | 4.25 → 4.66 | 4.70 → 4.87 | 11.2 → 5.6 |
| DeepSeek-V4-Flash | 73.1 → 81.9 | 64.7 → 77.5 | 81.4 → 86.4 | 23.5 → 7.0 | 4.04 → 4.90 | 4.66 → 4.98 | 10.4 → 2.7 |
| Gemini-3.7-Flash | 69.9 → 78.0 | 63.5 → 76.3 | 76.3 → 79.6 | 28.0 → 11.9 | 3.91 → 4.60 | 4.54 → 4.87 | 15.2 → 6.0 |
| GPT-5.6-Sol | 67.1 → 74.6 | 60.5 → 73.9 | 73.6 → 75.4 | 28.1 → 11.2 | 4.03 → 4.78 | 4.63 → 4.91 | 11.8 → 5.0 |
| Llama3.1-8B | 64.1 → 68.6 | 61.3 → 68.4 | 66.9 → 68.7 | 31.6 → 16.9 | 3.16 → 3.57 | 3.86 → 3.88 | 43.6 → 41.1 |
| Qwen3-32B | 68.4 → 80.7 | 62.5 → 78.7 | 74.2 → 82.6 | 28.2 → 9.4 | 3.49 → 4.18 | 4.27 → 4.72 | 25.8 → 15.7 |
| Qwen3-8B | 64.6 → 78.5 | 58.8 → 78.4 | 70.5 → 78.6 | 33.5 → 10.2 | 3.37 → 4.04 | 4.05 → 4.57 | 33.8 → 20.3 |
| Model macro-average | 68.1 → 76.7 | 62.0 → 74.9 | 74.1 → 78.5 | 27.2 → 11.0 | 3.75 → 4.39 | 4.39 → 4.69 | 21.7 → 13.8 |
All seven backbones achieved higher OC, lower OVR, and fewer omissions with CRA. Across models, OC increased from 68.1% to 76.7%, and OVR decreased from 27.2% to 11.0%. Both seniority ordering and comparisons involving Unrelated profiles improved. The profile-aware measures also improved on average, with KF rising from 3.75 to 4.39, BC from 4.39 to 4.69, and OMR falling from 21.7% to 13.8%.
These results extend the evidence across the entire set of tasks outside CapRoleBench-100, including every easy task. CRA’s improvements in ordering and profile-aware answer quality therefore extend beyond the diagnostic tasks selected using CPP performance.
Author Response to Reviewer wgE9 (Weaknesses 2-1 and Questions 3)
W2-1: CRA uses approximately 7.05 times the tokens and 4.90 times the cost of CPP, multiple LLM-generated intermediate states, and up to two self-revisions, but there is no equal-budget baseline using generic self-critique, longer prompting, structured career prompts, retrieval, or explicit refusal/authority control.
Q3: Material validation question: Can the authors compare CRA against equal-token or equal-cost baselines, including longer CPP prompting, generic self-reflection, structured career prompting, retrieval over the career history, and a simple profile-dependent authority/refusal controller?
A3: We added two baselines that incorporate these strategies and configured them using CRA’s token consumption as the budget target for each backbone. The evaluation covers seven backbones from six model families. Across these models, CRA achieved higher average Ordering Consistency (OC), lower Overreach Rate (OVR), and better profile-aware answer quality than both baselines.
- Direct evidence-and-scope prompting (DESP) extends CPP with explicit instructions to use career evidence, identify supported work, and stay within the role’s authority. It uses the additional budget to generate independent candidate answers and select the best one.
- The simpler contract-and-revision workflow (SCR) retrieves relevant career records, writes a compact contract specifying supported actions and authority boundaries, and applies generic critique and revision. Its critique checks for unsupported authority, unnecessary refusal, and omitted work the profile could perform.
These workflows incorporate the reviewer’s suggested longer prompting, structured use of career evidence, retrieval, profile-dependent scope control, and self-reflection. We calibrated candidate counts and revision limits on separate examples using token consumption, without consulting evaluation scores. We used Qwen3-32B as the evaluator for all three methods. Table 4 reports actual mean input-plus-output tokens across every inference call. Within each backbone, results use the same task–profile items completed by all three methods.
Table 4. Actual token consumption and ordering performance for the two simpler workflows and CRA. Values within each cell are ordered DESP / SCR / CRA. Token counts are per task–profile response.
| Backbone | Tokens per task–profile response | OC ↑ (%) | OVR ↓ (%) |
|---|---|---|---|
| Llama3.1-8B | 19,614 / 17,464 / 18,769 | 62.2 / 70.2 / 68.5 | 34.3 / 32.0 / 17.7 |
| Qwen3-8B | 13,677 / 11,572 / 12,683 | 70.3 / 66.3 / 77.3 | 26.7 / 30.7 / 14.3 |
| Qwen3-32B | 14,277 / 14,167 / 14,746 | 66.2 / 70.8 / 83.0 | 35.7 / 28.3 / 11.3 |
| Claude-Sonnet-5 | 21,640 / 7,981 / 24,450 | 68.4 / 69.9 / 72.4 | 33.0 / 25.9 / 13.1 |
| DeepSeek-V4-Flash | 12,795 / 9,292 / 14,482 | 55.6 / 65.4 / 81.8 | 49.3 / 30.7 / 11.3 |
| Gemini-3.7-Flash | 22,096 / 8,612 / 21,738 | 65.7 / 69.0 / 81.3 | 27.0 / 26.0 / 12.3 |
| GPT-5.6-Sol | 39,973 / 23,476 / 19,579 | 57.2 / 62.0 / 77.7 | 42.3 / 31.7 / 12.7 |
In nine of the 14 baseline comparisons, actual token use was within 12% of CRA’s. CRA achieved lower OVR in all nine and higher OC in eight. The exception was SCR on Llama3.1-8B, which achieved higher OC (70.2% vs. 68.5%) but also substantially higher OVR (32.0% vs. 17.7%). Across all seven backbones, CRA had lower OVR than both baselines, with paired task-bootstrap 95% confidence intervals excluding zero for all 14 differences. Its OC exceeded DESP’s on all seven backbones and SCR’s on six.
Table 5. Ordering performance and profile-aware answer quality, macro-averaged across the seven backbones.
| Method | OC ↑ (%) | S-OC ↑ (%) | B-OC ↑ (%) | OVR ↓ (%) | KF ↑ | BC ↑ | Omission rate ↓ (%) |
|---|---|---|---|---|---|---|---|
| DESP | 63.6 | 57.4 | 69.9 | 35.5 | 3.718 | 4.377 | 22.3 |
| SCR | 67.7 | 60.1 | 75.3 | 29.3 | 4.035 | 4.568 | 17.5 |
| CRA | 77.4 | 73.7 | 81.3 | 13.2 | 4.387 | 4.719 | 14.8 |
Across backbones, CRA improved OC by 13.8 percentage points over DESP and 9.7 points over SCR, while reducing OVR by 22.3 and 16.1 points, respectively. CRA also achieved higher Knowledge Fidelity and Behavior Coherence and fewer omissions than either baseline. These results support the value of CRA’s full workflow when compared with simpler methods that use additional inference to select answers or combine career retrieval, authority guidance, and critique and revision.
Author Response to Reviewer wgE9 (Weaknesses 1-1 and Questions 1)
We thank the reviewer for these comments.
A1:
CRA receives
task.target_esco_titleandtask.target_esco_codeas structured task fields. The controller compares these fields with the persona’s current occupation and career records and uses occupational relevance and alignment between task requirements and career evidence to select a behavioral regime. It does not receive Senior/Mid/Junior/Unrelated slot labels, audited profile ranks, gold pairwise outcomes, or evaluator scores. We will make this input interface explicit in the revision.We reran CPP on the same 100 tasks with the task metadata available to CRA. We inserted the following block immediately before
[Workplace Scenario]in the shared CPP user prompt.Table 1. Ordering performance after providing CPP with the task metadata available to CRA. Each entry shows original CPP → metadata-matched CPP → CRA. All values are percentages.
Adding the metadata alone left CPP’s average OC nearly unchanged (65.9% → 66.0%). CRA outperformed metadata-matched CPP in both OC and OVR on all 14 models. Across models, OC increased from 66.0% to 81.6%, and OVR decreased from 33.2% to 10.9%. The average gains covered both ordering among career stages (S-OC, 57.5% → 78.7%) and comparisons involving Unrelated profiles (B-OC, 74.6% → 84.4%).
Table 2. Profile-aware answer quality under the same task-information comparison, macro-averaged across 14 models.
CRA also retained higher Knowledge Fidelity and Behavior Coherence and a lower omission rate when CPP received the same task metadata. These results show that CRA’s advantage persists after matching access to the target occupation, ESCO code, and other task fields.