acceptodds
Under review as a conference paper at ICLR 2027

ACTOR: Action-Centered Temporal Organization for Literary Role-Playing Agents

Abstract

Large language agents for literary role-playing are typically modeled around dialogue generation, leaving characters' non-dialogue behaviors and their dynamic evolution over story time underrepresented. In this paper, we propose ACTOR (Action-Centered Temporal Organization for Literary Role-Playing Agents), an automated framework for constructing such behavior trajectories from raw novel texts. ACTOR extracts dialogue and persona-relevant non-dialogue behaviors, represents each behavior using a Thought–Action–Observation structure, and organizes behavior trajectories according to the internal temporal order of the story. We build a behavior dataset from 143 diverse literary works. For each source behavior, we generate counterfactual behaviors that are contextually plausible yet inconsistent with the character’s persona. We find that existing models struggle to reliably infer character-consistent actions from historical behaviors and evolving character states. Using these behavior pairs, we train models with Regularized Preference Optimization (RPO), which augments DPO with a negative log-likelihood loss on the source behavior to jointly optimize behavior preference and generation. We further design a four-way next-behavior selection task to evaluate the model’s ability to faithfully mimic character actions. Across 17,560 evaluation instances, the model trained on ACTOR’s preference pairs achieves substantially improved accuracy. This result demonstrates that structured, temporally organized supervision enables models to leverage character profiles and prior behaviors to infer subsequent actions, greatly advancing literary role-playing beyond mere dialogue imitation. Code and data are available at anonymity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
role-playing language agent, character behavior modeling, temporal behavior trajectories, preference optimization, next-action prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CLtCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ACTOR proposes an action-centered framework for improving literary role-playing agents by organizing character behavior into chronological Thought–Action–Observation (T-A-O) trajectories. The framework extracts character actions and contextual evidence from 143 literary works, reconstructs story chronology, maintains dynamic character profiles and memories, and generates plausible but character-inconsistent counterfactual behaviors. The resulting data are used for preference-based training with Relative Preference Optimization (RPO), combining DPO with an auxiliary negative log-likelihood objective.

The authors construct 270,856 training preference records and evaluate on 17,560 four-way next-behavior selection instances, including temporal evaluation on training works and cross-work evaluation on 28 additional works. The main contribution is an end-to-end data construction and training pipeline that emphasizes temporally grounded character actions rather than static character descriptions.

Critical strengths
  1. The paper goes beyond static character profiles by organizing behavior into chronological T-A-O records, reconstructing non-linear story timelines, and maintaining dynamic profiles. The resulting dataset is substantial, and the authors provide detailed prompts, schema descriptions, and structural validation statistics. This is a useful resource and engineering contribution for research on literary role-playing agents.

  2. Consistent improvements over the corresponding base models. ACTOR improves four-way next-behavior selection accuracy by approximately 8–9 percentage points for the evaluated Qwen-2.5 and Llama-3.1 backbones. The paper also reports gains in open-ended CoSER evaluations. These results suggest that the constructed supervision can improve model behavior under the authors' evaluation settings, even though the generality of the measured improvements remains uncertain.

Critical weaknesses

The main benchmark asks models to select the original behavior from four candidates: one source-text behavior and three automatically generated counterfactuals. Although the authors instruct the generator to make the alternatives plausible, character-inconsistent, and similar in length and form, this setup may still allow models to exploit artifacts of counterfactual generation or differences in how the options are phrased. High accuracy therefore does not necessarily demonstrate robust modeling of a character's motivations, long-range development, or behavior in genuinely unseen situations. Structural validation of the records does not establish the semantic validity of the counterfactuals. The open-ended evaluation provides additional evidence, but it relies on CoSER scores and LLM-based judging, and the qualitative examples still exhibit speaker-attribution and character-fidelity failures. Stronger independent validation is needed to support the central capability claim.

The optimization objective is a combination of standard DPO and an auxiliary likelihood loss. The main novelty consequently lies in the data construction and supervision design rather than a new learning algorithm. The reported gains are encouraging, but it remains unclear how much comes specifically from chronological T-A-O trajectories, counterfactual preference supervision, additional training exposure, or the auxiliary likelihood objective. In particular, the ablations indicate that removing long-term memory has little effect on Qwen models and can slightly improve Llama performance, which weakens the claim that long-range behavioral history is a central driver of the improvement. More controlled comparisons are needed to establish which components provide the substantive contribution.

Questions for authors
  1. Did the authors conduct a human evaluation of the generated counterfactuals to assess both their contextual plausibility and their inconsistency with the target character? If so, please report the annotation protocol, sample size, and agreement. Can the authors also provide results under controls that reduce the possibility of exploiting option-level artifacts, such as evaluating on independently authored alternatives or using a generation process different from the one used to construct the training data?

  2. How robust are the CoSER results to the choice of evaluator and judging model? Are there human evaluations or independent evaluation protocols that confirm the reported improvements, particularly in character fidelity and speaker attribution? This matters because the main multiple-choice benchmark may not fully measure open-ended role-playing ability.

  3. Can the authors provide controlled comparisons that separate the effects of T-A-O trajectory supervision, counterfactual preference training, and the auxiliary NLL objective? Given that removing long-term memory has little effect on some models, what evidence supports the claim that reconstructing and preserving long-range behavioral history is an important source of the gains?

  4. What is the provenance and licensing status of the 143 literary works, and what assessment was made of whether publicly releasing the derived behavior records is permitted under the applicable copyright and data-use terms?

4 comments

We thank the reviewer for the thoughtful feedback. We address the comments below.


Weaknesses 1: The main benchmark asks models to select the original behavior from four candidates: one source-text behavior and three automatically generated counterfactuals. Although the authors instruct the generator to make the alternatives plausible, character-inconsistent, and similar in length and form, this setup may still allow models to exploit artifacts of counterfactual generation or differences in how the options are phrased. High accuracy therefore does not necessarily demonstrate robust modeling of a character's motivations, long-range development, or behavior in genuinely unseen situations. Structural validation of the records does not establish the semantic validity of the counterfactuals. & Questions 1:

  • Questions 1.1: Did the authors conduct a human evaluation of the generated counterfactuals to assess both their contextual plausibility and their inconsistency with the target character? If so, please report the annotation protocol, sample size, and agreement.
  • Questions 1.2: Can the authors also provide results under controls that reduce the possibility of exploiting option-level artifacts, such as evaluating on independently authored alternatives or using a generation process different from the one used to construct the training data?

1.1:

We selected 400 evaluation questions through stratified random sampling. Two annotators independently reviewed all questions, with a third adjudicating disagreements. The assessment covered four dimensions: source fidelity, checking whether the original behavior matched the source text; contextual validity, checking consistency between the character context and the current scene, as well as potential future-information leakage; counterfactual quality, assessing the contextual plausibility of each alternative; and answer discriminability, checking whether the provided character information clearly supported the original behavior more strongly than the other options.

The final results are as follows:

Evaluation DimensionQuestions PassingPass Rate
Source Fidelity398/40099.50%
Contextual Validity394/40098.50%
Counterfactual Quality (all three alternatives pass)379/40094.75%
Answer Discriminability386/40096.25%
All Four Criteria Met375/40093.75%

1.2:

We added an options-only evaluation to assess model performance based solely on the answer options. The results are shown below:

ModelBaseAfter RPOGain (percentage points)
Qwen-2.5-7B38.29%44.12%+5.83
Qwen-2.5-14B45.77%52.56%+6.79
Llama-3.1-8B36.64%39.25%+2.61

Models may learn differences between the options during training. However, the gains observed when sufficient character information is provided are substantially larger than those obtained from the options alone.


Weaknesses 1: The open-ended evaluation provides additional evidence, but it relies on CoSER scores and LLM-based judging, and the qualitative examples still exhibit speaker-attribution and character-fidelity failures. Stronger independent validation is needed to support the central capability claim. & Questions 2:

  • How robust are the CoSER results to the choice of evaluator and judging model? Are there human evaluations or independent evaluation protocols that confirm the reported improvements, particularly in character fidelity and speaker attribution? This matters because the main multiple-choice benchmark may not fully measure open-ended role-playing ability.

To reduce scoring variability, we used a strong frontier model as the judge and averaged the results across repeated evaluation runs.

We also conducted a human evaluation by randomly sampling 10 shared scenarios and anonymously pairing each of the three ACTOR models with its corresponding backbone. Three annotators independently rated the conversations on a 1–5 scale across four dimensions: storyline consistency, anthropomorphism, character fidelity, and storyline quality. They also identified errors such as character identity confusion and speaking on behalf of other characters, and indicated their overall preferences:

ModelNon-empty Character TurnsError RateEmpty Turns After Cleaning
Qwen-2.5-7B41/15626.28%5
ACTOR-Qwen-2.5-7B16/14311.19%0
Qwen2.5-14B18/11116.22%3
ACTOR-Qwen-2.5-14B7/977.22%0
Llama-3.1-8B64/14045.71%0
ACTOR-Llama-3.1-8B18/15411.69%0

Given substantial variation in annotators’ use of the rating scale, we did not report absolute human scores. Instead, we used their judgments to qualitatively assess whether ACTOR models generally outperformed their backbones and included illustrative case analyses in the appendix.


Weaknesses 2: The optimization objective is a combination of standard DPO and an auxiliary likelihood loss. The main novelty consequently lies in the data construction and supervision design rather than a new learning algorithm. The reported gains are encouraging, but it remains unclear how much comes specifically from chronological T-A-O trajectories, counterfactual preference supervision, additional training exposure, or the auxiliary likelihood objective. In particular, the ablations indicate that removing long-term memory has little effect on Qwen models and can slightly improve Llama performance, which weakens the claim that long-range behavioral history is a central driver of the improvement. More controlled comparisons are needed to establish which components provide the substantive contribution. & Questions 3:

  • Questions 3.1: Can the authors provide controlled comparisons that separate the effects of T-A-O trajectory supervision, counterfactual preference training, and the auxiliary NLL objective?
  • Questions 3.2: Given that removing long-term memory has little effect on some models, what evidence supports the claim that reconstructing and preserving long-range behavioral history is an important source of the gains?

3.1:

We first explored SFT by fine-tuning Qwen2.5-7B-Instruct solely on the ground-truth behavior text. The results on the anonymous evaluation set are shown below:

ModelSelection accuracyProportion of A predictions
Qwen2.5-7B-Instruct71.34%27.57%
SFT step50070.31%27.93%
SFT step100063.93%39.65%
SFT step150058.25%47.23%

SFT did not improve behavior-selection accuracy under this configuration. A possible explanation is that SFT encourages the generation of correct behavior without explicitly distinguishing it from incorrect alternatives in relation to the character context. Continued fine-tuning may also interfere with the generation behavior inherited from the backbone.

DPO and RPO instead introduce comparative supervision through counterfactual negative examples. Both achieved substantially higher behavior-selection accuracy than the backbone:

ModelSelection accuracy
Backbone71.34%
Qwen2.5-7B-DPO87.34%
Qwen2.5-7B-RPO80.20%

Although pure DPO achieved higher selection accuracy, its performance under the CoSER evaluation framework was considerably lower than that of RPO. Some DPO simulations also exhibited repetitive generation and outputs that reached the token limit. One possible explanation is that DPO optimizes the relative preference between chosen and rejected responses without directly maximizing the likelihood of the chosen response. Improved preference discrimination therefore does not necessarily translate into better open-ended generation.

ModelCoSER mean score
Backbone10.43
Qwen2.5-7B-DPO10.7375
Qwen2.5-7B-RPO21.99

Considering both behavior selection and open-ended interaction, we chose RPO as our final training objective. Its auxiliary NLL term directly supervises the generation of correct behavior while retaining counterfactual preference learning.


3.2:

As discussed in the ablation section of our paper, dynamic profiles summarize a character’s past experiences and personality and may partially compensate for the absence of long-term memory. To further examine this explanation, we conducted an additional evaluation using static profiles. We selected a subset of questions for which the characters’ initial profiles could be recovered and used these initial versions as the static profiles. The results are shown below:

ModelAll memoryWithout long-term memoryChange (percentage points)
ACTOR-Qwen-2.5-7B80.31%80.13%−0.18
ACTOR-Qwen-2.5-14B88.37%87.09%−1.28
ACTOR-Llama-3.1-8B77.57%76.71%−0.86

With static profiles, removing long-term memory reduced accuracy across all three models.


Questions 4:

  • What is the provenance and licensing status of the 143 literary works, and what assessment was made of whether publicly releasing the derived behavior records is permitted under the applicable copyright and data-use terms?

All the book texts we use are sourced from publicly available versions. We do not publish the original texts, and we will reconfirm copyright permissions before releasing the constructed dataset.


We hope that our explanations above can clarify your doubts and you can consider our work more favorably.

Reviewer_esTGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ACTOR, an automated framework for constructing character-centered behavior trajectories from literary works. The framework extracts dialogue and persona-relevant non-dialogue behaviors into Thought–Action–Observation units, reconstructs their in-story chronological order, identifies consequential behaviors, and updates character profiles when sustained changes occur. Each source behavior is paired with generated alternatives intended to be contextually plausible but inconsistent with the target character. These pairs support training with Regularized Preference Optimization, combining DPO with a negative log-likelihood loss on the source behavior.

The resulting corpus covers 143 literary works and includes 270,856 training preference records. Evaluation comprises 17,560 four-way next-behavior questions, covering both later portions of training-book trajectories and 28 works excluded from fine-tuning. Across three backbones, ACTOR training improves selection accuracy by approximately nine percentage points. The paper additionally reports higher average scores in open-ended, multi-turn interaction under the CoSER evaluation framework, although improvements vary across evaluation dimensions.

Critical strengths
  1. The paper offers a useful formulation and a substantial resource for character behavior modeling.
    The explicit representation of both verbal and non-verbal decisions, together with evolving profiles and structured behavioral history, provides a valuable foundation for studying character consistency. Distinguishing the chronology of events from their order of presentation is particularly relevant to literary sources containing flashbacks or nonlinear narration.

  2. The improvements are substantial and consistent across backbones, with encouraging transfer evidence.
    Table 1 shows improvements from 71.34% to 80.20% for Qwen-2.5-7B, 76.96% to 86.00% for Qwen-2.5-14B, and 69.15% to 78.04% for Llama-3.1-8B. Gains of 7.04–8.89 percentage points also remain on works excluded from ACTOR training, supporting the usefulness of the supervision beyond continuation of familiar training trajectories.

Critical weaknesses
  1. The benchmark has insufficient validation that success requires character-specific understanding, rather than exploiting the construction of the alternatives.
    The correct option is extracted from the source text, whereas all incorrect options are generated by the same construction pipeline used to produce training negatives. Matching length, register, and structural form is a useful precaution, but it does not establish that the alternatives are equally plausible in the situation or free from systematic semantic cues. Consequently, improvements could partly reflect learning to distinguish source behaviors from the negative-generation distribution.

  2. The distinctive benefit of temporal organization is not isolated, and the claimed temporal information boundary requires stronger verification.
    The experiments establish that training on ACTOR-derived data improves performance, but do not isolate the contribution of reconstructing story chronology or maintaining dynamic character states. There are no matched comparisons against original narrative order, static profiles, or simpler same-data training objectives such as SFT and DPO. The existing context-removal experiments are not direct tests of these construction choices. In particular, removing long-term memory reduces accuracy by only 0.50 and 0.37 percentage points for the two Qwen models and improves Llama by 0.19 points (Figure 3). Profiles may summarize some historical information, as the authors acknowledge, so this result does not invalidate temporal modeling. However, it also does not establish what temporal reconstruction contributes.

Questions for authors
  1. How much of the benchmark’s accuracy depends on character information?
    Could you report options-only, scene-only, and mismatched-profile/history controls, together with a blinded semantic audit assessing source fidelity, situational plausibility, persona inconsistency, and answer ambiguity? Results on a human-validated subset with independently authored alternatives would materially strengthen the benchmark’s validity.

  2. What is the incremental contribution of temporal reconstruction and dynamic profiles?
    Under matched data and training budgets, how do original narrative order and static profiles compare with the complete pipeline? How do SFT, DPO, and RPO compare? Please also report how often timeline reconstruction changes the original order and whether the benefit is concentrated on those cases.

  3. How are temporal information boundaries and split isolation enforced?
    When predicting at time tt, is the evidence used for key-behavior verification and memory selection restricted to what has occurred by tt? Have you audited shared scenes and source spans across different characters’ training and evaluation records? Please also clarify any sampling or filtering steps that reconcile the stated 80/20 split with 270,856 training records and only 11,770 temporal-evaluation questions.

3 comments

We thank the reviewer for the thoughtful feedback. We address the comments below.


Weaknesses 1: The benchmark has insufficient validation that success requires character-specific understanding, rather than exploiting the construction of the alternatives.The correct option is extracted from the source text, whereas all incorrect options are generated by the same construction pipeline used to produce training negatives. Matching length, register, and structural form is a useful precaution, but it does not establish that the alternatives are equally plausible in the situation or free from systematic semantic cues. Consequently, improvements could partly reflect learning to distinguish source behaviors from the negative-generation distribution. & Questions 1:

  • How much of the benchmark’s accuracy depends on character information?Could you report options-only, scene-only, and mismatched-profile/history controls, together with a blinded semantic audit assessing source fidelity, situational plausibility, persona inconsistency, and answer ambiguity? Results on a human-validated subset with independently authored alternatives would materially strengthen the benchmark’s validity.

Building upon our existing experiments, we supplemented additional settings where only options or only scenarios were provided. The results are shown below:

ModelOptions-onlyScene-only
Qwen2.5-7B Base38.29%51.29%
ACTOR-Qwen-2.5-7B44.12%63.57%
Qwen2.5-14B Base45.77%61.58%
ACTOR-Qwen-2.5-14B52.56%69.92%
Llama3.1-8B Base36.64%49.70%
ACTOR-Llama-3.1-8B39.25%62.48%

The selection accuracy of each model in these two scenarios is far lower than that when character information is provided.

And we also supplemented a comparative experiment using the initial profile of the character with the influence of long-term memory removed.

ModelStatic profileDynamic profileGain from dynamic profiles
ACTOR-Qwen-2.5-7B80.13%81.99%+1.86
ACTOR-Qwen-2.5-14B87.09%87.57%+0.48
ACTOR-Llama-3.1-8B76.71%79.83%+3.11

We selected 400 evaluation questions through stratified random sampling. Two annotators independently reviewed all questions, with a third adjudicating disagreements. The assessment covered four dimensions: source fidelity, checking whether the original behavior matched the source text; contextual validity, checking consistency between the character context and the current scene, as well as potential future-information leakage; counterfactual quality, assessing the contextual plausibility of each alternative; and answer discriminability, checking whether the provided character information clearly supported the original behavior more strongly than the other options.

The final results are as follows:

Evaluation DimensionQuestions PassingPass Rate
Source Fidelity398/40099.50%
Contextual Validity394/40098.50%
Counterfactual Quality (all three alternatives pass)379/40094.75%
Answer Discriminability386/40096.25%
All Four Criteria Met375/40093.75%

Weaknesses 2: The distinctive benefit of temporal organization is not isolated, and the claimed temporal information boundary requires stronger verification.The experiments establish that training on ACTOR-derived data improves performance, but do not isolate the contribution of reconstructing story chronology or maintaining dynamic character states. There are no matched comparisons against original narrative order, static profiles, or simpler same-data training objectives such as SFT and DPO. The existing context-removal experiments are not direct tests of these construction choices. In particular, removing long-term memory reduces accuracy by only 0.50 and 0.37 percentage points for the two Qwen models and improves Llama by 0.19 points (Figure 3). Profiles may summarize some historical information, as the authors acknowledge, so this result does not invalidate temporal modeling. However, it also does not establish what temporal reconstruction contributes. & Questions 2:

  • Questions 2.1: What is the incremental contribution of temporal reconstruction and dynamic profiles? Under matched data and training budgets, how do original narrative order and static profiles compare with the complete pipeline?

  • Questions 2.2: How do SFT, DPO, and RPO compare?

  • Questions 2.3: Please also report how often timeline reconstruction changes the original order and whether the benefit is concentrated on those cases.

    2.1:

Timeline reordering aims to maintain temporal consistency in the data, aligning character profiles and memories with the character’s state at the target time and reducing the risk of including future experiences in evaluation inputs.

Furthermore, to assess the value of dynamic profiles, we conducted an additional paired inference-time ablation, replacing only the dynamic character profiles with their initial profiles. Both conditions excluded long-term memory to control for its potential compensatory effect. We evaluated questions for which the initial profiles were recoverable and differed from the corresponding dynamic profiles. The results are shown below:

ModelStatic profileDynamic profileGain from dynamic profiles
ACTOR-Qwen2.5-7B80.13%81.99%+1.86
ACTOR-Qwen2.5-14B87.09%87.57%+0.48
ACTOR-Llama3.1-8B76.71%79.83%+3.11

Under identical model and evaluation conditions, dynamic profiles achieved higher accuracy across all three models.

2.2:

We first explored SFT by fine-tuning Qwen2.5-7B-Instruct solely on the ground-truth behavior text. The results on the anonymous evaluation set are shown below:

ModelSelection accuracyProportion of A predictions
Qwen2.5-7B-Instruct71.34%27.57%
SFT step50070.31%27.93%
SFT step100063.93%39.65%
SFT step150058.25%47.23%

SFT did not improve behavior-selection accuracy under this configuration. A possible explanation is that SFT encourages the generation of correct behavior without explicitly distinguishing it from incorrect alternatives in relation to the character context. Continued fine-tuning may also interfere with the generation behavior inherited from the backbone.

DPO and RPO instead introduce comparative supervision through counterfactual negative examples. Both achieved substantially higher behavior-selection accuracy than the backbone:

ModelSelection accuracy
Backbone71.34%
Qwen-2.5-7B-DPO87.34%
Qwen-2.5-7B-RPO80.20%

Although pure DPO achieved higher selection accuracy, its performance under the CoSER evaluation framework was considerably lower than that of RPO. Some DPO simulations also exhibited repetitive generation and outputs that reached the token limit. One possible explanation is that DPO optimizes the relative preference between chosen and rejected responses without directly maximizing the likelihood of the chosen response. Improved preference discrimination therefore does not necessarily translate into better open-ended generation.

ModelCoSER mean score
Backbone10.43
Qwen-2.5-7B-DPO10.7375
Qwen-2.5-7B-RPO21.99

Considering both behavior selection and open-ended interaction, we chose RPO as our final training objective. Its auxiliary NLL term directly supervises the generation of correct behavior while retaining counterfactual preference learning.


2.3:

Our pipeline processes timelines in two stages: initial detection and review. Across the 143 works, the initial detection stage generated 99 candidate moves and flagged 76 records with uncertain temporal ordering. Following LLM review, 61 candidate segments were reordered. A subsequent manual audit identified only nine additional valid candidates that had been missed.

A temporal reordering reduces the risk of future information leaking into historical memory, helping ensure the validity of behavior prediction evaluations rather than directly increasing accuracy. Without such adjustments, models may answer using information about events that have not yet occurred, making it difficult to distinguish inference based on character context from answer leakage. Temporal reordering also supports the generation of temporally consistent dynamic profiles. Our experiments demonstrate the value of dynamic profiles for behavior prediction.

Furthermore, performance improvements also involve trajectory preference training, so the accuracy gains cannot be attributed entirely to temporal reordering.


Questions 3:

  • Questions 3.1: How are temporal information boundaries and split isolation enforced? When predicting at time (t), is the evidence used for key-behavior verification and memory selection restricted to what has occurred by (t)? Have you audited shared scenes and source spans across different characters’ training and evaluation records?
  • Questions 3.2: Please also clarify any sampling or filtering steps that reconcile the stated 80/20 split with 270,856 training records and only 11,770 temporal-evaluation questions.

3.1:

At prediction time, recent and long-term memories include only experiences preceding the target action in the reconstructed timeline, and the latest profile version preceding the target scene is used. Key-behavior verification, however, is not restricted to past events: subsequent plot developments are used as evidence of a behavior’s impact.

The 80/20 split is performed independently within each character’s trajectory, so scene-level isolation across characters is not guaranteed. However, because each training example predominantly contains information about its target character, the likelihood of training–evaluation leakage is extremely low. We also provide a complementary cross-work evaluation in which the training and evaluation works are entirely disjoint.


3.2:

The 80/20 split described in the paper applies to each character’s action sequence before final sample construction: the first 80% forms the training candidate pool, and the remaining 20% forms the within-work temporal-generalization evaluation candidate pool. The two pools then undergo different filtering procedures.

For training, we require each character to have a valid profile and at least five actions in their complete trajectory. We exclude the first action and actions with invalid counterfactuals. All training candidates that pass these checks are retained.

For evaluation, in addition to the action-validity checks above, we require each character’s complete trajectory to contain at least 20 actions and their profile to pass the usability check, excluding characters with overly sparse behavioral histories or insufficient characterization. We then apply further sampling: for each character in each scene, we randomly retain at most one key behavior and one non-key behavior. We also cap each character’s contribution at 15 key behaviors and five non-key behaviors. These redundancy-reduction and sampling constraints help reduce repeated scoring of similar character interactions within a scene and limit the dominance of a few characters with disproportionately many actions in the overall metric.


We hope that our explanations above can clarify your doubts and you can consider our work more favorably.

Reviewer_7YHdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ACTOR, which extracts dialogue and persona-relevant non-dialogue behaviors from 143 literary works, represents them in a Thought–Action–Observation format, and organizes them according to the reconstructed story chronology. The authors construct preference training data using dynamic profiles, historical memories, and counterfactual behaviors, and train models with RPO. Evaluation covers within-work temporal generalization, cross-work behavior selection, and open-ended interactions in CoSER.

Critical strengths

The behavior data construction pipeline is systematic and has potential for reuse. It combines behavior extraction, verification against textual evidence, temporal organization, dynamic profiles, and counterfactual construction to provide structured supervision for modeling literary characters’ behaviors. The consistent improvements in behavior selection across three backbone models also support the practical utility of this data.

Critical weaknesses

W1. The independent contribution of the core temporal design has not been sufficiently established.

My understanding is that the paper’s main innovation relative to CoSER and DREAM lies in the value of reconstructing the actual story chronology, compared with existing ways of organizing character information. However, the current ablations primarily remove the profile and memory components. They neither directly compare the original narrative order with the reconstructed chronological order nor quantify how many scenes actually require reordering.

W2. Key information about the semantic validity of the counterfactual benchmark and the data split is still missing, please see Questions for details.

Questions for authors

Q1. How many works, scenes, or behaviors are actually reordered by the temporal correction process, and how is the accuracy of these corrections validated? Could the authors compare the original narrative order with the reconstructed chronological order while keeping the backbone, training data size, and optimization objective fixed, and separately report results on the subset that genuinely requires reordering?

Q2. How many instances were manually checked, and by how many annotators? Did the assessment separately evaluate whether the alternatives are plausible in context, inconsistent with the character’s persona, and whether the provided context uniquely identifies the original behavior? How does an 80/20 split for each character yield 270,856 training records and 11,770 within-work temporal test questions? Is there additional filtering or sampling, or does a single behavior correspond to multiple records?

2 comments

We thank the reviewer for the thoughtful feedback. We address the comments below.


Weaknesses 1: The independent contribution of the core temporal design has not been sufficiently established.My understanding is that the paper’s main innovation relative to CoSER and DREAM lies in the value of reconstructing the actual story chronology, compared with existing ways of organizing character information. However, the current ablations primarily remove the profile and memory components. They neither directly compare the original narrative order with the reconstructed chronological order nor quantify how many scenes actually require reordering. & Questions 1:

  • Questions 1.1: How many works, scenes, or behaviors are actually reordered by the temporal correction process, and how is the accuracy of these corrections validated?
  • Questions 1.2: Could the authors compare the original narrative order with the reconstructed chronological order while keeping the backbone, training data size, and optimization objective fixed, and separately report results on the subset that genuinely requires reordering?

1.1:

Our pipeline processes timelines in two stages: initial detection and review. Across the 143 works, the initial detection stage generated 99 candidate moves and flagged 76 records with uncertain temporal ordering. Following LLM review, 61 candidate segments were reordered. A subsequent manual audit identified only nine additional valid candidates that had been missed.

For example, in My Ántonia, one correction moved a meeting explicitly dated to 1908 after the earlier college-summer events narrated subsequently in the text. In Wuthering Heights, Isabella’s letter explicitly states, “I came last night,” establishing that the first-night experiences described in the letter must precede the act of writing it the following day.


1.2:

And timeline reordering aims to maintain temporal consistency in the data, aligning character profiles and memories with the character’s state at the target time and reducing the risk of including future experiences in evaluation inputs, rather than directly pursuing higher accuracy. In fact, an incorrect order may leak future information and artificially inflate accuracy.

For example, in Strange Case of Dr Jekyll and Mr Hyde, the narrative first depicts Lanyon’s psychological collapse and refusal to discuss Jekyll, and only later recounts, through his manuscript, how he witnessed Hyde’s transformation and cried out in horror. If the original narrative order is used directly, Lanyon’s subsequent fear and rejection of Jekyll may be included in his historical memory when predicting his reaction that night, prematurely revealing answer-relevant information.

So our validation instead focuses on checking event chronology and the correctness of repositioning against the source text rather than on predictive accuracy during evaluation.


Weaknesses 2: Key information about the semantic validity of the counterfactual benchmark and the data split is still missing, please see Questions for details. & Questions 2:

  • Questions 2.1: How many instances were manually checked, and by how many annotators? Did the assessment separately evaluate whether the alternatives are plausible in context, inconsistent with the character’s persona, and whether the provided context uniquely identifies the original behavior?
  • Questions 2.2: How does an 80/20 split for each character yield 270,856 training records and 11,770 within-work temporal test questions? Is there additional filtering or sampling, or does a single behavior correspond to multiple records?

2.1:

We selected 400 evaluation questions through stratified random sampling. Two annotators independently reviewed all questions, with a third adjudicating disagreements. The assessment covered four dimensions: source fidelity, checking whether the original behavior matched the source text; contextual validity, checking consistency between the character context and the current scene, as well as potential future-information leakage; counterfactual quality, assessing the contextual plausibility of each alternative; and answer discriminability, checking whether the provided character information clearly supported the original behavior more strongly than the other options.

For example, one question from A Doll’s House asks how Nora would respond when her husband forbids her to leave. The counterfactuals offer three contextually feasible strategies: bargaining for financial security, seeking reconciliation through appeasement, and submitting to his authority. However, the provided information about her current state indicates that she now rejects her husband’s control, giving stronger support to the original behavior of firmly asserting her departure. This example illustrates how the task assesses changes in a character’s state.


2.2:

The 80/20 split described in the paper applies to each character’s action sequence before final sample construction: the first 80% forms the training candidate pool, and the remaining 20% forms the within-work temporal-generalization evaluation candidate pool. The two pools then undergo different filtering procedures.

For training, we require each character to have a valid profile and at least five actions in their complete trajectory. We exclude the first action and actions with invalid counterfactuals. All training candidates that pass these checks are retained.

For evaluation, in addition to the action-validity checks above, we require each character’s complete trajectory to contain at least 20 actions and their profile to pass the usability check, excluding characters with overly sparse behavioral histories or insufficient characterization. We then apply further sampling: for each character in each scene, we randomly retain at most one key behavior and one non-key behavior. We also cap each character’s contribution at 15 key behaviors and five non-key behaviors. These redundancy-reduction and sampling constraints help reduce repeated scoring of similar character interactions within a scene and limit the dominance of a few characters with disproportionately many actions in the overall metric.


We hope that our explanations above can clarify your doubts and you can consider our work more favorably.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your reviews of this submission. The authors have posted detailed responses to the concerns behind your weak-reject ratings, and I would appreciate it if you could take a moment to read them ASAP. Please let us know whether the rebuttal addresses your main concerns, and whether it changes your assessment in either direction. If issues remain, a brief note on what is still missing would be very helpful, both for the final decision and for the authors.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.