acceptodds
Under review as a conference paper at ICLR 2027

LiveAnyRoleBench: Towards Role-playing as Any Entity

Abstract

Large language models (LLMs) are increasingly used to simulate diverse roles, from fictional characters in games to real-world entities in social simulations. Yet existing benchmarks largely focus on few well-known characters (e.g., fictional protagonists or historical figures), who already appear in pretraining data, so even high scores may be contaminated by memorization instead of reflecting general simulation ability. Moreover, evaluating only on human characters can yield biased results, leaving performance consistency across fundamentally different entity types (e.g., warships, space missions, or animals) unclear. To address both gaps, we build role-playing tasks directly from Wikipedia, which is a continually updated source that covers an enormous range of entities. By converting each entity article into a sequence of dated actions, we introduce LiveAnyRoleBench, which evaluates whether an LLM can role-play as any entity by predicting its next action given its own history actions as context. This is, to our knowledge, the first role-playing benchmark that dates every task and spans arbitrary entity types (137 types, 25,208 entities, 969,088 actions). Evaluating 13 models, we find that every model's score drops by 3 to 18 percentage points on actions dated after its training cutoff. Across entity types, the strongest model leads almost everywhere, but among models of similar strength, rankings on human characters do not predict rankings on other types. Grounded in real-world actions, LiveAnyRoleBench provides a foundation for role-playing agents and world simulators requiring fine-grained interaction and complex decision-making. We release the dataset, construction pipeline, and all model outputs.

Primary area:
datasets and benchmarks
Keywords:
role-playing agents, large language models, world simulators, data contamination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2GSCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends role-playing evaluation to a wider range of entities, including companies and warships. LiveAnyRoleBench converts Wikipedia articles into dated action sequences. Models generate the next action from ten previous actions and a topic-specific question. The dataset covers 25,208 entities across 137 types. Experiments compare 13 models across entity types and before and after their training cutoffs, aiming to separate memorization from behavior prediction.

Critical strengths
  1. The research question is interesting. Including companies, warships, and animals broadens role-playing evaluation. The large dataset and reusable pipeline could be useful for studying behavior prediction across entity types.

  2. The writing is generally easy to follow. Figure 1 clearly explains the data-processing pipeline and evaluation setup.

Critical weaknesses
  1. The context may not provide enough information for reliable next-action evaluation.

a) An entity's next decision may depend on market conditions, government policies, or technological developments that are not included in its ten previous actions. The pipeline also skips some intermediate actions. The paper acknowledges that the target may not be the next action in reality, but the effect of missing these actions on prediction remains unclear. The remaining ordering errors in the small reference set also leave some uncertainty about the context–target sequence in the evaluation data.

b) The model generates an action rather than chooses from fixed options. Even with the topic specified, several actions may be plausible given the history. Although the judge allows partial credit, it scores predictions against the single action that actually occurred. A low score may therefore reflect missing information rather than poor prediction ability. Further evidence that the provided context helps distinguish the observed action from other plausible actions would strengthen the evaluation.

  1. Further analysis would help clarify what the results show about role-playing ability.

a) The entity-level aggregation and cutoff checks are useful. My main concern is whether the scores reflect entity-specific behavior prediction beyond domain knowledge.

b) Figure 4 shows rank changes among models of similar strength. However, it does not show uncertainty in these ranks. Section 3.4 reports pairwise intervals for human-character scores, but these do not establish whether the rank changes across entity types are reliable. The results suggest that models with similar human-character scores can differ on other types. A stronger claim that human-character ranks do not predict other ranks would benefit from direct statistical support.

c) These differences may also reflect domain knowledge or general event prediction. Comparisons that remove the entity's identity, remove its history, or keep only the topic would help clarify how much entity-specific information contributes to the score. Figure 3 usefully shows lower scores after the training cutoff and checks whether newer actions are harder. The analysis could offer deeper insight into what this gap reveals about role-playing ability beyond familiarity with historical events.

Questions for authors
  1. Could you clarify how the evaluation handles other reasonable next actions? Is there evidence that the provided context helps identify the observed action, and have you assessed how skipping intermediate actions affects prediction?

  2. Could you provide uncertainty estimates for the cross-type rank changes in Figure 4? Do you have comparisons that remove identity, remove history, or keep only the topic to help separate entity-specific prediction from domain knowledge?

7 comments
AuthorsOpenReview ↗

Author's Response (1/5): Evaluation Validity: Part 1 Missing Actions and Context-Target Correspondence

Thank you for your thoughtful review and constructive suggestions, especially on the recognition of the interesting research question, the broader role-playing evaluation, the large dataset and reusable pipeline, and the clarity of Figure 1.

Following your suggestions, we have measured how missing actions and ordering errors in the tasks affect the results, credited other plausible next actions, compared inputs that remove the identity, remove the history or keep only the topic, and analyzed where the score drop at the training cutoff comes from, and tested the cross-type rank changes on more entities.

We have uploaded a revised version and the corresponding additions and clarifications are marked in blue in the manuscript. We hope these updates will address your concerns on the current submission and look forward to further discussion with you.


Evaluation Validity: Part 1 Missing Actions and Context-Target Correspondence (W1a, W1b; Q1)

Two types of missing actions. Each timeline is a longest chain of actions whose order is certain (Section 2.2), so certain actions with proper dates are still left off the timeline. Two types of these off-timeline actions lie between the last context action and the end of the target's date range: (1) skipped actions end before the target starts, while (2) overlapping actions overlap the target's dates.

Missing actions in context. Only skipped actions can be added to the context, because they certainly end before the target starts. Among all 20,109 evaluated tasks, 1,809 tasks (9.0%) contain at least one, usually exactly one. We added them after the ten context actions in time order and evaluated four models of different strength on the affected 1,809 tasks. Scores changed by −1.2 to +0.7 points, and the four models kept their order.

Missing plausible target actions. Both skipped actions and overlapping actions can serve as plausible alternative targets. Among all tasks, 8,230 tasks (40.9%) contain at least one. We labelled all these potential alternatives with the plannability label used for targets and kept the plannable ones (60.2%, close to 61.3% on the timeline), found in 6,667 tasks (33.2%). The judge then scored the 13 models' original predictions against each alternative as an additional correct answer, with all other inputs unchanged. Across all tasks, this raises the 13 models by only 1.0 to 1.5 points (3.9 to 5.5 on the 6,667 tasks it affects), and no pair of models changes order.

ModelAdd missing actions in context (1,809 tasks)Add missing plausible targets (6,667 tasks)
GPT-6 Astra−0.1+4.6
DeepSeek-V4-Pro+0.7+5.5
GPT-4o+0.6+4.6
Qwen3-8B−1.2+4.0

Context-Target Correspondence. Several next actions can be plausible given the same history, yet each prediction is currently scored only against the single action that appears on the main timeline, so a low score may not reflect poor prediction. We test this on the 1,000 sampled tasks (one per entity) used for the input comparisons in Comment 3/5, 246 of which have a plannable alternative next action. We change the context in two ways: (1) history removal tests the model with or without the ten context actions; (2) history replacement tests the model with the entity's own ten actions or those of another entity of the same type. We then score each model's prediction against the original target and against each alternative target respectively. A prediction hits a target when it receives tier 2 or higher (per our rubrics, tier 2 means the prediction describes the same action as the target). We count a prediction as an alternative-only hit when it hits at least one alternative but not the original target.

Result. (1) Removing the history lowers the hit rate on the original target by 7.7 to 17.5 points, but changes the alternative-only hit rate by just −3.7 to +2.4 points, with no consistent direction. (2) Replacing the history with another entity's gives the same trend: the original target loses 26.5 points on average, while the alternative-only hit rate falls by only 1.8 points on average.

This suggests that the ten context actions do carry information that helps distinguish the original target actions from any alternative ones. The table below gives the change in hit rate (percentage points) on these 246 tasks when the context history is removed, and the value in parentheses is the change when the context is replaced by another entity's history.

ModelHits on the original targetHits on an alternative only
GPT-6 Astra−11.8 (−46.7)+1.2 (+2.4)
DeepSeek-V4-Pro−11.8 (−26.0)+2.4 (−2.8)
GPT-4o−17.5 (−19.5)−3.7 (−3.3)
Qwen3-8B−7.7 (−13.8)−1.6 (−3.7)

Section 3.5 now summarizes these three tests, and Appendix J gives the full results (Tables 15 to 17).

AuthorsOpenReview ↗

Author's Response (2/5): Evaluation Validity: Part 2 Temporal Ordering Error

Evaluation Validity: Part 2 Temporal Ordering Error (W1a)

Two types of inconsistencies. Temporal inconsistency can appear among the ten context actions (intra-context inconsistency), or between context and target action (context-target inconsistency). The 23.9% reported in Section 2.2 is calculated from 11-action windows drawn from three reference articles. Each window has ten context actions followed by one target action, the same layout as a task. Of all 360 verified windows, 18.6% contain only intra-context reversals and 5.3% contain a context-target reversal.

(1) Intra-context inconsistency

Construct a subset with strong intra-context inconsistency. We sampled 1,000 tasks (one per entity), of which 877 had context actions sharing a date. We put these date-sharing actions in a new random order. Whatever their original order was, about 50% of the pairs among them are now in the wrong order, so this shuffle deliberately creates strong intra-context inconsistency.

Results. We show that this type has little influence on model performance. We re-ran four models of different strength (GPT-6 Astra, DeepSeek-V4-Pro, GPT-4o and Qwen3-8B). Across three runs with the original prompts, mean scores varied by −1.4 to +1.4 points. Scores with reordered prompts differed by −1.0 to +1.5 points, comparable to the variation between original runs. By contrast, on the 1,000 sampled tasks, removing all ten context actions lowers the scores by 5.6 to 14.3 points, so the models do use the context.

(2) Context-target inconsistency

For next-action prediction, what matters more is whether the target action happens after the ten context actions. Therefore, although only 5.3% of the windows contain this type, we measure its impact in detail below.

Construct a cleaner subset with no context-target inconsistency. From all 20,109 evaluated tasks (all 1k entities), we selected a cleaner subset of 4,180 tasks (21%, 615 entities) in which every action has an explicit date that puts the target strictly after its context actions. The subset covers all 10 domains and 132 of all 137 entity types.

Result. We show that this type also has little influence on our main conclusions about model ranking and cutoff drops. Each cell below gives the value "on all tasks" → "on the subset".

ModelOverall RankingOverall ScoreCutoff Drops
GPT-6 Astra1 → 155.1 → 57.318.2 → 19.6
DeepSeek-V4-Pro4 → 540.9 → 41.86.9 → 12.5
GPT-4o7 → 735.9 → 38.29.6 → 12.0
Qwen3-8B11 → 1226.1 → 27.23.0 → 3.2
  • Training cutoff. All 13 models still score lower after their training cutoff.
  • Ranking. The 13 models keep their order by average score, except two pairs of neighbours: DeepSeek-V4-Pro and GPT-6 Luna (0.3 points apart), and Qwen3-8B and Llama-3.1-8B (0.8 points apart).
  • Scores rise slightly on this subset. This is mainly due to which entities are included, rather than the filtering for clean tasks. When all their tasks are considered, the 615 entities in this subset already score 1.2 to 2.2 points higher than the full set. This subset contains only the clean tasks of these entities, yielding a similar and reasonable score increase of 0.6 to 2.4 points.

Section 2.2 now splits the 23.9% into the two types, and Section 3.5 reports both analyses, with the table above as Table 3. Appendix I gives the per-model results and the reordering test. Outside factors (W1a). The ten context actions are not limited to the entity's own decisions: they can also include what other parties did and outside events that the article reports (Appendix J gives an example), and adding information beyond the article is left as future work.

AuthorsOpenReview ↗

Author's Response (3/5): Role-playing Ability: Part 1 General Knowledge vs. Entity-specific Information

As in your review, we understand role-playing ability as the ability to perform "entity-specific behavior prediction". A score on such a task can come from two other sources: (1) general knowledge (domain knowledge or general event prediction), which needs no information about the specific entity; (2) familiarity with the target actions, which a model may have read in training.

We measure how much each contributes. Part 1 (this comment) changes the model input to separate general knowledge from entity-specific information. Part 2 (next comment) analyzes the gap at each model's training cutoff. Part 3 (Comment 5/5) tests whether the rank changes across entity types in Figure 4 are reliable.

Role-playing Ability: Part 1 General Knowledge vs. Entity-specific Information (W2a, W2c; Q2)

Experiment. The table shows the five input settings we test. They differ in two ways: whether the entity's name is given (rows), and which history (ten context actions) is given (columns). We test them on the 1,000 sampled tasks (one per entity) with the four models used in the previous comments.

Without historyWith its own historyWith other's history
With nameHistory removedFull input—
Without nameTopic onlyName removedHistory replaced
  • Full input: the input used in the paper. It has the entity's name, Wiki class, its ten context actions and the question.
  • Name removed (your "remove identity" request): the full input with the name replaced by [ENTITY].
  • History removed (your "remove history" request): the full input without the ten actions. Only the date of the last one is kept.
  • Topic only (your "keep only the topic" request): no name and no history, and the question is reduced to its topic ("What happened next regarding [topic]?"). The model sees only the Wikidata class (e.g., "enterprise"), the date of the last context action and the topic.
  • History replaced (our addition): the name-removed input with the ten actions of another entity of the same type (the one closest in time). It checks whether any history of the same type helps as much as the entity's own.

Results. All predictions are scored by the same judge against the original target. Each cell is the mean score on these 1,000 tasks, so the full-input column is close to, but not identical to, Table 2 in paper.

ModelFull inputName removedHistory removedTopic onlyHistory replaced
GPT-6 Astra56.754.742.45.3*18.6
DeepSeek-V4-Pro40.438.326.116.019.3
GPT-4o34.532.922.813.816.4
Qwen3-8B23.623.418.013.414.4

* GPT-6 Astra refused to answer on about 790 of the 1,000 topic-only prompts (for example, it asks which entity is meant), so the comparisons with "topic only" below omit GPT-6 Astra.

Conclusions.

  • General knowledge alone still gives 13 to 16 points. When only the topic is given, model can only predict using general domain knowledge or general event prediction, and their scores range from 13.4 to 16.0 points.
  • Entity-specific information accounts for the other 10.2 to 24.4 points, as shown by the score drop from "Full input" to "Topic only". This includes both entity name and entity history, but history matters more. Models can often recover the entity name from the provided history, so removing the name alone has less impact than removing the history.
  • It has to be the entity's own history. Scores with replaced history drop to a level comparable to "Topic only" scores (only 1.0 to 3.3 points higher), which means the replaced history provides similar amount of information as general knowledge does, and that the entity's own history does contribute a lot to predicting the right next target action. The same result is shown in Comment 1/5.
  • Model ranking. Entity-specific information is what separates the models. In the first three columns of the table (where entity-specific information is still present), the four models keep the same order and are 24.4 to 33.1 points apart from first to last. However, in the last two columns the models are only 2.6 to 4.9 points apart from first to last, and with replaced history GPT-6 Astra falls behind DeepSeek-V4-Pro.

Summary. General knowledge is the shared base of role-playing ability, and the entity's own history is what mainly decides a model's performance. The models get about the same 13 to 16 points from general knowledge, but the points from entity-specific information grow from 10.2 for Qwen3-8B to 24.4 for DeepSeek-V4-Pro, and for the stronger models they exceed what general knowledge gives. Section 3.6 now reports this comparison (Table 4), and Appendix K defines the five inputs and shows them for one task.

Reviewer_3uM6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LiveAnyRoleBench, which is a dataset curated via Wikipedia articles to simulate any entity (fictional characters, real-world entities) and shows that many AI models' performances drop on evals corresponding to entities dated after the model's training cutoff date. This reveals a bias in that the existing "high scores" reported by model providers may stem from having pretraining data contaminated by memorization instead of revealing a more nuanced understanding of the rule. The dataset curated here is also quite large (137 types, 25,208 entities, 969,088 actions), and the evaluations are extensive (covering 13 models), enabling evaluations of LLMs in very new and complex decision-making environments.

Critical strengths

This is the first role-playing benchmark that dates every task and spans arbitrary entity types (using a cool Wikipedia-based idea to gather this). As such, it is highly scalable and can test "out-of-distribution" entities.

Critical weaknesses

It isn't clear how the pipeline performs for entities whose Wikipedia article is sparse. For example, some articles may span only 1 paragraph but may not reflect actual interaction with the entity, though I could be wrong. Is there a way to quantify the variance/uncertainty of the Wikipedia distribution from a more "real" distribution? I also think this affects the quality of the underlying dataset. Is there a way to quantify how this affected the distribution/variance of actions, and in turn, is there a measure of the variance of the final performance of the models that were evaluated?

Questions for authors

I have a couple of questions:

  1. In this work, GPT-4.1-mini was used to clean the data / label the next actions etc., but GPT-4.1-mini is not really a strong model. Was there human evaluation etc. done to verify that the plannable/unplannable labels were real and not hallucinated/incorrect (or is there a way to get a reasoning trace to justify this by selecting a stronger model)?
  2. Can the authors also compute the steerability of LLMs towards these roles? What I mean by this is the following: each role corresponds to a steering vector vv, but instead of adding vv, we can add αv\alpha v for some α>0\alpha>0. Ideally, the larger α\alpha is, the more representative the underlying model should be of the role, until α\alpha becomes "too large" and the model's performance collapses (see reference 1). The largest possible α\alpha under which performance is maintained, the steerability, might be a good metric to include in the evaluation (or at least in the discussion for future work) for these corresponding roles.
  3. Can the authors provide some examples of the dialogs (for maybe 2-3 roles) in the appendix to assess the richness of the interactions?

References:

Role Steering of Language Models for Social Simulations Song et. al. (2026)

2 comments
AuthorsOpenReview ↗

Author's Response: Data Construction & Future Directions

Comment: Thank you for your thoughtful review and constructive suggestions, especially on the recognition of the first role-playing benchmark that dates every task and spans arbitrary entity types, and its scalable Wikipedia-based construction. Following your suggestions, we have checked the plannable labels with three annotators, compared scores across article lengths, and added three complete task examples. We have uploaded a revised version and the corresponding additions and clarifications are marked in blue in the manuscript.


Data Construction (W1; Q1)

Sparse articles. Entities are sampled systematically along article length within each class, so the sample keeps the length distribution of the full entity pool (Section 2.2). Only entities with at least eleven timeline actions yield tasks (Section 2.3). Scores vary little with article length (Section 3.5, Appendix M): when the 1,000 evaluated entities are split into four groups by article length, from 6–34 sentences to 108–756, each model's four group means lie within 5.3 points of each other (within 2.5 points for nine of the 13 models), and the model order in every group closely follows the overall order (Kendall's τ of 0.92 to 1.00). Each model's score comes with a 95% entity-bootstrap interval of at most ±1.3 points (Table 9). Comparison with fuller records of the same entities, such as news archives, is now a future direction in the Conclusion.

Plannable labels. The plannable labels are assigned by GPT-4.1, while GPT-4.1-mini extracts the dated actions (Section 2.2). They agree with the labels of one of the authors on 79 to 81 of 90 actions across three runs. Following your suggestion, three annotators who are not authors also checked the labels on 200 further actions: all three give the same label to 159 of them (Fleiss' κ = 0.72), and their majority confirms 84 of the 100 task targets (Section 2.2, Appendix L).

Future Directions (Q2, Q3)

Steerability. Thank you for this suggestion. Measuring steerability, by scoring open-weight models on our tasks under increasing steering strengths, is now a future direction in the Conclusion, citing Song et al. [1].

Task examples and multi-turn interaction. Each task is a single prompt rather than a dialogue, answered with one sentence describing the entity's next action (Section 3.1). Following your suggestion, Appendix N now gives three complete tasks, from a company, an animal and a human character, each with the full prompt, the target and four models' predictions with the judge's tiers. Multi-turn interaction is now a future direction in the Conclusion.


Thank you again for your careful reading and constructive suggestions. We fully respect the time and effort you have devoted to this review and will remain actively engaged throughout the entire discussion period. We would be glad to hear your further thoughts and work through any remaining questions with you.

References: [1] Role Steering of Language Models for Social Simulations. 2026.

AuthorsOpenReview ↗

Author's Follow-Up

Dear Reviewer 3uM6,

With the November 18 rebuttal deadline approaching, we would greatly appreciate your updated thoughts on our response and the revised manuscript. Following your suggestions, we have conducted additional experiments, as summarized at the beginning of our response. We have uploaded a revised version and the corresponding additions and clarifications are marked in blue in the manuscript.

If any point would benefit from further clarification, please let us know, and we will respond promptly.

Thank you again for your thoughtful review and constructive suggestions.

Best regards, The Authors

Reviewer_BnN1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce LiveAnyRoleBench, a role-playing benchmark that broadens role-playing evaluation beyond human characters to diverse non-human entities. To construct LiveAnyRoleBench, the authors extract actions from Wikipedia, reconstruct date-labeled, entity-specific timelines, and produce over 400k next-action prediction tasks spanning 137 entity types. The experiments evaluate 13 LLMs. One notable observation is that all evaluated LLMs perform worse on actions that occurred after their respective training cutoffs. Moreover, models with similar performance on human role-playing tasks perform differently across non-human entity types. To evaluate role-playing ability across these entity types, the authors use an LLM judge to compare predicted actions with the recorded next actions and assign scores on a five-tier scale ranging from 0 to 100.

Critical strengths

S1. Broad coverage of non-human entities. The benchmark covers diverse entities, including companies, universities, ships, space missions, and animals. This distinctive setup enables comparisons beyond human-centered role-playing and reveals differences in model performance across diverse entity types.

S2. Potential utility of a large-scale benchmark with temporal information. The Wikipedia-based construction pipeline combines entity-specific action histories with temporal information, supporting evaluation relative to individual models’ training cutoffs. The authors also note that the benchmark can be updated using newer Wikipedia dumps.

Critical weaknesses

W1. Limited analysis of evaluation reliability. The authors report over 80% agreement between the LLM judge and human judgments provided by a single author. However, reliance on a single author leaves potential annotator bias unexamined, and the limited number of validated samples provides insufficient evidence of the reliability of the LLM judge for evaluation. Validation with more independent annotators and a broader sample would help establish the reliability of the LLM judge.

W2. Remaining inconsistencies in temporal ordering. The paper reports that some action sequences in the validation sample still contradict the reference timeline, even after applying the proposed ordering method. Such inconsistencies may disrupt the temporal relationship between context actions and targets, affecting the validity of next-action prediction tasks. Since the paper proposes a new benchmark dataset with dated, entity-specific timelines as a key contribution, further reducing these inconsistencies and assessing their impact would strengthen its practical utility.

Questions for authors

Q1. Could the authors extend the judge validation to a broader sample of entity types with multiple independent annotators, and report inter-annotator agreement?

Q2. Could the authors assess how correcting or filtering temporal inconsistencies affects the main evaluation results?

4 comments
AuthorsOpenReview ↗

Author's Response (1/2): Evaluation Reliability and Human Annotation

Thank you for your thoughtful review and constructive suggestions, especially on the recognition of the broad coverage of non-human entities and the dated timelines that support evaluation relative to each model's training cutoff.

Following your suggestions, we have validated the judge with three independent annotators on a broader sample covering all entity types, and measured how the remaining ordering errors affect the main results.

We have uploaded a revised version and the corresponding additions and clarifications are marked in blue in the manuscript. We hope these updates will address your concerns on the current submission and look forward to further discussion with you.


Evaluation Reliability and Human Annotation (W1; Q1)

Broadened annotators. We increase the number of annotators from one to three. All three are computer science students interested in role-playing, and none is an author of this paper. Each annotator independently labelled all 300 predictions using the five-tier definitions in Section 3.1. For each prediction, they saw the same information as the judge, which includes the target, the question, the last two context actions, and the prediction. They were not shown the judge's answer or any model information.

More samples. We sampled 300 new predictions from the evaluation split, with 30 from each of the 10 domains. These cover all 137 entity types and 275 entities, with 23 or 24 predictions from each of the 13 evaluated models. None of them overlap with the 200 predictions previously annotated by the author.

Results. All three annotators give the same tier to 211 of the 300 predictions (70.3%; Fleiss' κ = 0.70), and pairs of annotators agree on 75.7%, 79.3% and 83.0% of them. The judge agrees with the majority vote on 219 predictions (73.0%) and is within one tier of it on 251. When the judge disagreed with the majority, it assigned a higher tier in 35 cases and a lower tier in 38, showing no clear directional bias. The other 8 predictions have no majority tier.

Section 3.1 and Conclusion now report this validation. Appendix E now details the sampling, the annotation protocol, the full judge-majority tier table and the agreement in each of the 10 domains.

AuthorsOpenReview ↗

Author's Response (2/2): Temporal Ordering Errors and Their Impact

Temporal Ordering Errors and Their Impact (W2; Q2)

Two types of inconsistencies. Temporal inconsistency can appear among the ten context actions (intra-context inconsistency), or between context and target action (context-target inconsistency). The 23.9% reported in Section 2.2 is calculated from 11-action windows drawn from three reference articles. Each window has ten context actions followed by one target action, the same layout as a task. Of all the verified 360 windows, 18.6% contain only intra-context reversals and 5.3% contain a context-target reversal.

(1) Intra-context inconsistency

Construct a subset with strong intra-context inconsistency. We sampled 1,000 tasks (one per entity), of which 877 had context actions sharing a date. We put these date-sharing actions in a new random order. Whatever their original order was, about 50% of the pairs among them are now in the wrong order, so this shuffle deliberately creates strong intra-context inconsistency.

Results. We show that this type has little influence on model performance. We re-ran four models of different strength (GPT-6 Astra, DeepSeek-V4-Pro, GPT-4o and Qwen3-8B). Across three runs with the original prompts, mean scores varied by −1.4 to +1.4 points. Scores with reordered prompts differed by −1.0 to +1.5 points, comparable to the variation between original runs. By contrast, on the 1,000 sampled tasks, removing all ten context actions lowers the scores by 5.6 to 14.3 points, so the models do use the context.

(2) Context-target inconsistency

For next-action prediction, what matters more is whether the target action happens after the ten context actions. Therefore, although only 5.3% of the windows contain this type, we measure its impact in detail below.

Construct a cleaner subset with no context-target inconsistency. From all 20,109 evaluated tasks (all 1k entities), we selected a cleaner subset of 4,180 tasks (21%, 615 entities) in which every action has an explicit date that puts the target strictly after its context actions. The subset covers all 10 domains and 132 of all 137 entity types.

Result. We show that this type also has little influence on our main conclusions about model ranking and cutoff drops. Full results are in Appendix I. Each cell below gives the value "on all tasks" → "on the subset".

ModelOverall RankingOverall ScoreCutoff Drops
GPT-6 Astra1 → 155.1 → 57.318.2 → 19.6
DeepSeek-V4-Pro4 → 540.9 → 41.86.9 → 12.5
GPT-4o7 → 735.9 → 38.29.6 → 12.0
Qwen3-8B11 → 1226.1 → 27.23.0 → 3.2
  • Training cutoff. All 13 models still score lower after their training cutoff.
  • Ranking. The 13 models keep their order by average score, except two pairs of neighbours: DeepSeek-V4-Pro and GPT-6 Luna (0.3 points apart), and Qwen3-8B and Llama-3.1-8B (0.8 points apart).
  • Scores rise slightly on this subset. This is mainly due to which entities are included, rather than the filtering for clean tasks. When all their tasks are considered, the 615 entities in this subset already score 1.2 to 2.2 points higher than the full set. This subset contains only the clean tasks of these entities, yielding a similar and reasonable score increase of 0.6 to 2.4 points.

Section 2.2 now splits the 23.9% into the two types, and Section 3.5 reports both analyses, with the table above as Table 3. Appendix I gives the per-model results and the reordering test.

Thank you again for your careful reading and constructive suggestions. Your suggestions have helped make the judge validation more complete and the effect of ordering errors explicit. We hope you find the fuller discussions in the revised manuscript helpful as you revisit these points. We fully respect the time and effort you have devoted to this review and will remain actively engaged throughout the entire discussion period. We would be glad to hear your further thoughts and work through any remaining questions with you.

AuthorsOpenReview ↗

Author's Follow-Up

Dear Reviewer BnN1,

With the November 18 rebuttal deadline approaching, we would greatly appreciate your updated thoughts on our response and the revised manuscript. Following your suggestions, we have conducted additional experiments, as summarized at the beginning of our response. We have uploaded a revised version and the corresponding additions and clarifications are marked in blue in the manuscript.

If any point would benefit from further clarification, please let us know, and we will respond promptly.

Thank you again for your thoughtful review and constructive suggestions.

Best regards, The Authors

Thank you for the authors' effort for a detailed rebuttal. it mitigates most of my original concerns.

However, I expect a benchmark dataset to be cleaner. The ordering errors still remain and I expect them to persist each time the benchmark is updated.

I also share Reviewer 2GSC’s remaining concern about the interpretation of the prediction scores.

Therefore, I maintain my rating of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.