- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper extends role-playing evaluation to a wider range of entities, including companies and warships. LiveAnyRoleBench converts Wikipedia articles into dated action sequences. Models generate the next action from ten previous actions and a topic-specific question. The dataset covers 25,208 entities across 137 types. Experiments compare 13 models across entity types and before and after their training cutoffs, aiming to separate memorization from behavior prediction.
Critical strengths
-
The research question is interesting. Including companies, warships, and animals broadens role-playing evaluation. The large dataset and reusable pipeline could be useful for studying behavior prediction across entity types.
-
The writing is generally easy to follow. Figure 1 clearly explains the data-processing pipeline and evaluation setup.
Critical weaknesses
- The context may not provide enough information for reliable next-action evaluation.
a) An entity's next decision may depend on market conditions, government policies, or technological developments that are not included in its ten previous actions. The pipeline also skips some intermediate actions. The paper acknowledges that the target may not be the next action in reality, but the effect of missing these actions on prediction remains unclear. The remaining ordering errors in the small reference set also leave some uncertainty about the context–target sequence in the evaluation data.
b) The model generates an action rather than chooses from fixed options. Even with the topic specified, several actions may be plausible given the history. Although the judge allows partial credit, it scores predictions against the single action that actually occurred. A low score may therefore reflect missing information rather than poor prediction ability. Further evidence that the provided context helps distinguish the observed action from other plausible actions would strengthen the evaluation.
- Further analysis would help clarify what the results show about role-playing ability.
a) The entity-level aggregation and cutoff checks are useful. My main concern is whether the scores reflect entity-specific behavior prediction beyond domain knowledge.
b) Figure 4 shows rank changes among models of similar strength. However, it does not show uncertainty in these ranks. Section 3.4 reports pairwise intervals for human-character scores, but these do not establish whether the rank changes across entity types are reliable. The results suggest that models with similar human-character scores can differ on other types. A stronger claim that human-character ranks do not predict other ranks would benefit from direct statistical support.
c) These differences may also reflect domain knowledge or general event prediction. Comparisons that remove the entity's identity, remove its history, or keep only the topic would help clarify how much entity-specific information contributes to the score. Figure 3 usefully shows lower scores after the training cutoff and checks whether newer actions are harder. The analysis could offer deeper insight into what this gap reveals about role-playing ability beyond familiarity with historical events.
Questions for authors
-
Could you clarify how the evaluation handles other reasonable next actions? Is there evidence that the provided context helps identify the observed action, and have you assessed how skipping intermediate actions affects prediction?
-
Could you provide uncertainty estimates for the cross-type rank changes in Figure 4? Do you have comparisons that remove identity, remove history, or keep only the topic to help separate entity-specific prediction from domain knowledge?
7 comments
Author's Response (2/5): Evaluation Validity: Part 2 Temporal Ordering Error
Evaluation Validity: Part 2 Temporal Ordering Error (W1a)
Two types of inconsistencies. Temporal inconsistency can appear among the ten context actions (intra-context inconsistency), or between context and target action (context-target inconsistency). The 23.9% reported in Section 2.2 is calculated from 11-action windows drawn from three reference articles. Each window has ten context actions followed by one target action, the same layout as a task. Of all 360 verified windows, 18.6% contain only intra-context reversals and 5.3% contain a context-target reversal.
(1) Intra-context inconsistency
Construct a subset with strong intra-context inconsistency. We sampled 1,000 tasks (one per entity), of which 877 had context actions sharing a date. We put these date-sharing actions in a new random order. Whatever their original order was, about 50% of the pairs among them are now in the wrong order, so this shuffle deliberately creates strong intra-context inconsistency.
Results. We show that this type has little influence on model performance. We re-ran four models of different strength (GPT-6 Astra, DeepSeek-V4-Pro, GPT-4o and Qwen3-8B). Across three runs with the original prompts, mean scores varied by −1.4 to +1.4 points. Scores with reordered prompts differed by −1.0 to +1.5 points, comparable to the variation between original runs. By contrast, on the 1,000 sampled tasks, removing all ten context actions lowers the scores by 5.6 to 14.3 points, so the models do use the context.
(2) Context-target inconsistency
For next-action prediction, what matters more is whether the target action happens after the ten context actions. Therefore, although only 5.3% of the windows contain this type, we measure its impact in detail below.
Construct a cleaner subset with no context-target inconsistency. From all 20,109 evaluated tasks (all 1k entities), we selected a cleaner subset of 4,180 tasks (21%, 615 entities) in which every action has an explicit date that puts the target strictly after its context actions. The subset covers all 10 domains and 132 of all 137 entity types.
Result. We show that this type also has little influence on our main conclusions about model ranking and cutoff drops. Each cell below gives the value "on all tasks" → "on the subset".
| Model | Overall Ranking | Overall Score | Cutoff Drops |
|---|---|---|---|
| GPT-6 Astra | 1 → 1 | 55.1 → 57.3 | 18.2 → 19.6 |
| DeepSeek-V4-Pro | 4 → 5 | 40.9 → 41.8 | 6.9 → 12.5 |
| GPT-4o | 7 → 7 | 35.9 → 38.2 | 9.6 → 12.0 |
| Qwen3-8B | 11 → 12 | 26.1 → 27.2 | 3.0 → 3.2 |
- Training cutoff. All 13 models still score lower after their training cutoff.
- Ranking. The 13 models keep their order by average score, except two pairs of neighbours: DeepSeek-V4-Pro and GPT-6 Luna (0.3 points apart), and Qwen3-8B and Llama-3.1-8B (0.8 points apart).
- Scores rise slightly on this subset. This is mainly due to which entities are included, rather than the filtering for clean tasks. When all their tasks are considered, the 615 entities in this subset already score 1.2 to 2.2 points higher than the full set. This subset contains only the clean tasks of these entities, yielding a similar and reasonable score increase of 0.6 to 2.4 points.
Section 2.2 now splits the 23.9% into the two types, and Section 3.5 reports both analyses, with the table above as Table 3. Appendix I gives the per-model results and the reordering test. Outside factors (W1a). The ten context actions are not limited to the entity's own decisions: they can also include what other parties did and outside events that the article reports (Appendix J gives an example), and adding information beyond the article is left as future work.
Author's Response (3/5): Role-playing Ability: Part 1 General Knowledge vs. Entity-specific Information
As in your review, we understand role-playing ability as the ability to perform "entity-specific behavior prediction". A score on such a task can come from two other sources: (1) general knowledge (domain knowledge or general event prediction), which needs no information about the specific entity; (2) familiarity with the target actions, which a model may have read in training.
We measure how much each contributes. Part 1 (this comment) changes the model input to separate general knowledge from entity-specific information. Part 2 (next comment) analyzes the gap at each model's training cutoff. Part 3 (Comment 5/5) tests whether the rank changes across entity types in Figure 4 are reliable.
Role-playing Ability: Part 1 General Knowledge vs. Entity-specific Information (W2a, W2c; Q2)
Experiment. The table shows the five input settings we test. They differ in two ways: whether the entity's name is given (rows), and which history (ten context actions) is given (columns). We test them on the 1,000 sampled tasks (one per entity) with the four models used in the previous comments.
| Without history | With its own history | With other's history | |
|---|---|---|---|
| With name | History removed | Full input | — |
| Without name | Topic only | Name removed | History replaced |
- Full input: the input used in the paper. It has the entity's name, Wiki class, its ten context actions and the question.
- Name removed (your "remove identity" request): the full input with the name replaced by
[ENTITY]. - History removed (your "remove history" request): the full input without the ten actions. Only the date of the last one is kept.
- Topic only (your "keep only the topic" request): no name and no history, and the question is reduced to its topic ("What happened next regarding [topic]?"). The model sees only the Wikidata class (e.g., "enterprise"), the date of the last context action and the topic.
- History replaced (our addition): the name-removed input with the ten actions of another entity of the same type (the one closest in time). It checks whether any history of the same type helps as much as the entity's own.
Results. All predictions are scored by the same judge against the original target. Each cell is the mean score on these 1,000 tasks, so the full-input column is close to, but not identical to, Table 2 in paper.
| Model | Full input | Name removed | History removed | Topic only | History replaced |
|---|---|---|---|---|---|
| GPT-6 Astra | 56.7 | 54.7 | 42.4 | 5.3* | 18.6 |
| DeepSeek-V4-Pro | 40.4 | 38.3 | 26.1 | 16.0 | 19.3 |
| GPT-4o | 34.5 | 32.9 | 22.8 | 13.8 | 16.4 |
| Qwen3-8B | 23.6 | 23.4 | 18.0 | 13.4 | 14.4 |
* GPT-6 Astra refused to answer on about 790 of the 1,000 topic-only prompts (for example, it asks which entity is meant), so the comparisons with "topic only" below omit GPT-6 Astra.
Conclusions.
- General knowledge alone still gives 13 to 16 points. When only the topic is given, model can only predict using general domain knowledge or general event prediction, and their scores range from 13.4 to 16.0 points.
- Entity-specific information accounts for the other 10.2 to 24.4 points, as shown by the score drop from "Full input" to "Topic only". This includes both entity name and entity history, but history matters more. Models can often recover the entity name from the provided history, so removing the name alone has less impact than removing the history.
- It has to be the entity's own history. Scores with replaced history drop to a level comparable to "Topic only" scores (only 1.0 to 3.3 points higher), which means the replaced history provides similar amount of information as general knowledge does, and that the entity's own history does contribute a lot to predicting the right next target action. The same result is shown in Comment 1/5.
- Model ranking. Entity-specific information is what separates the models. In the first three columns of the table (where entity-specific information is still present), the four models keep the same order and are 24.4 to 33.1 points apart from first to last. However, in the last two columns the models are only 2.6 to 4.9 points apart from first to last, and with replaced history GPT-6 Astra falls behind DeepSeek-V4-Pro.
Summary. General knowledge is the shared base of role-playing ability, and the entity's own history is what mainly decides a model's performance. The models get about the same 13 to 16 points from general knowledge, but the points from entity-specific information grow from 10.2 for Qwen3-8B to 24.4 for DeepSeek-V4-Pro, and for the stronger models they exceed what general knowledge gives. Section 3.6 now reports this comparison (Table 4), and Appendix K defines the five inputs and shows them for one task.
Author's Response (1/5): Evaluation Validity: Part 1 Missing Actions and Context-Target Correspondence
Thank you for your thoughtful review and constructive suggestions, especially on the recognition of the interesting research question, the broader role-playing evaluation, the large dataset and reusable pipeline, and the clarity of Figure 1.
Following your suggestions, we have measured how missing actions and ordering errors in the tasks affect the results, credited other plausible next actions, compared inputs that remove the identity, remove the history or keep only the topic, and analyzed where the score drop at the training cutoff comes from, and tested the cross-type rank changes on more entities.
We have uploaded a revised version and the corresponding additions and clarifications are marked in blue in the manuscript. We hope these updates will address your concerns on the current submission and look forward to further discussion with you.
Evaluation Validity: Part 1 Missing Actions and Context-Target Correspondence (W1a, W1b; Q1)
Two types of missing actions. Each timeline is a longest chain of actions whose order is certain (Section 2.2), so certain actions with proper dates are still left off the timeline. Two types of these off-timeline actions lie between the last context action and the end of the target's date range: (1) skipped actions end before the target starts, while (2) overlapping actions overlap the target's dates.
Missing actions in context. Only skipped actions can be added to the context, because they certainly end before the target starts. Among all 20,109 evaluated tasks, 1,809 tasks (9.0%) contain at least one, usually exactly one. We added them after the ten context actions in time order and evaluated four models of different strength on the affected 1,809 tasks. Scores changed by −1.2 to +0.7 points, and the four models kept their order.
Missing plausible target actions. Both skipped actions and overlapping actions can serve as plausible alternative targets. Among all tasks, 8,230 tasks (40.9%) contain at least one. We labelled all these potential alternatives with the plannability label used for targets and kept the plannable ones (60.2%, close to 61.3% on the timeline), found in 6,667 tasks (33.2%). The judge then scored the 13 models' original predictions against each alternative as an additional correct answer, with all other inputs unchanged. Across all tasks, this raises the 13 models by only 1.0 to 1.5 points (3.9 to 5.5 on the 6,667 tasks it affects), and no pair of models changes order.
Context-Target Correspondence. Several next actions can be plausible given the same history, yet each prediction is currently scored only against the single action that appears on the main timeline, so a low score may not reflect poor prediction. We test this on the 1,000 sampled tasks (one per entity) used for the input comparisons in Comment 3/5, 246 of which have a plannable alternative next action. We change the context in two ways: (1) history removal tests the model with or without the ten context actions; (2) history replacement tests the model with the entity's own ten actions or those of another entity of the same type. We then score each model's prediction against the original target and against each alternative target respectively. A prediction hits a target when it receives tier 2 or higher (per our rubrics, tier 2 means the prediction describes the same action as the target). We count a prediction as an alternative-only hit when it hits at least one alternative but not the original target.
Result. (1) Removing the history lowers the hit rate on the original target by 7.7 to 17.5 points, but changes the alternative-only hit rate by just −3.7 to +2.4 points, with no consistent direction. (2) Replacing the history with another entity's gives the same trend: the original target loses 26.5 points on average, while the alternative-only hit rate falls by only 1.8 points on average.
This suggests that the ten context actions do carry information that helps distinguish the original target actions from any alternative ones. The table below gives the change in hit rate (percentage points) on these 246 tasks when the context history is removed, and the value in parentheses is the change when the context is replaced by another entity's history.
Section 3.5 now summarizes these three tests, and Appendix J gives the full results (Tables 15 to 17).