acceptodds
Under review as a conference paper at ICLR 2027

CEPO: Causal Entropy Policy Optimization for Long-Form Narrative Generation

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in open-ended text generation; however, generating ultra-long novels with long-range causal dependencies, stable character personalities, and coherent narrative arcs remains a substantial challenge. The standard autoregressive generation process is prone to accumulating local inconsistencies, and conventional sequence-level reinforcement learning (e.g., GRPO) suffers from severe credit assignment problems when applied to long-form texts comprising tens of thousands of tokens. To address these challenges, we propose a hierarchical agent-based framework for novel generation that decomposes long-form writing into multiple verifiable stages, utilizing synopses, blueprints, a memory manager, and a Narrative World Graph. Furthermore, to tackle reward sparsity in the optimization of extended texts, we introduce a dense token-level reward mechanism guided by an LLM evaluator. This mechanism translates qualitative narrative feedback into a causal separation of coherent prefixes and contaminated suffixes. Building upon this mechanism, we propose Causal Entropy Policy Optimization (CEPO). By combining suffix-level penalties with an entropy-guided modulation mechanism, CEPO provides a heuristic approximation of token-level contributions around localized failure points, thereby effectively reducing the variance and bias of credit assignment in the generation of ultra-long narratives. Extensive automated and human evaluations on a large-scale dataset of online web novels demonstrate that, despite requiring significantly fewer parameters, the proposed method is competitive with strong baselines on six evaluated dimensions. Consequently, this work establishes an effective paradigm for the high-quality alignment and generation of complex, ultra-long narratives.

Primary area:
reinforcement learning
Keywords:
Ultra-long text generation, Hierarchical agent-based framework, Long-range coherence.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1z9qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a hierarchical multi-agent framework for long-form novel generation, combining planning, memory, world modeling, and critique. It further proposes CEPO, which uses an LLM critic to locate the first structural error and assigns suffix-level, entropy-modulated rewards to train the Writer Agent. Experiments on web-novel data show improvements over several RL baselines in automated and human evaluations.

Critical strengths

The paper addresses a practically relevant challenge in long-form generation, namely sparse and delayed feedback caused by error propagation over long contexts. The proposed reward design is relatively simple and interpretable, and can potentially be integrated into existing training pipelines. In addition, the empirical analysis before the method provides some motivation for the proposed error-localized reward scheme.

Critical weaknesses
  • Unclear motivation and narrative. The introduction presents the multi-agent framework, RL training, and sparse-reward problem almost simultaneously, without establishing a clear logical chain explaining why each component is necessary. As a result, the paper appears to introduce a complex system before clearly defining the underlying problem.

  • Unbalanced contribution framing. The multi-agent framework is largely an engineering scaffold, and its individual components do not appear particularly novel. However, the paper presents the framework and CEPO as comparably important contributions, which may lead readers to overestimate the novelty of the overall system. More controlled comparisons are needed to isolate the contribution of CEPO.

  • Questionable assumptions in the reward design. CEPO relies on an LLM critic to identify the first error and assumes that subsequent tokens are primarily affected by this error. These assumptions may be unreliable, since the critic can misidentify the error location, later content may recover from an earlier mistake, and multiple independent errors may occur in the same sequence.

  • Overclaiming the granularity and causal interpretation. The proposed reward remains a suffix-level signal determined by a single error boundary, rather than genuinely fine-grained token-level credit assignment. Moreover, terms such as “causal” appear stronger than what is supported by the method, since the paper does not establish a causal relationship between the identified error and all subsequent tokens. The entropy-based weighting also seems largely heuristic.

  • Narrow target problem and incomplete experimental correspondence. The method focuses mainly on consistency degradation after an early error, while novel generation involves many other dimensions such as plot quality, characterization, style, and readability. In addition, the experiments do not fully isolate or directly validate the proposed mechanism, making it difficult to determine whether the reported gains are specifically due to CEPO or to other components of the framework.

Questions for authors

please see weakness above

5 comments

[W1][W2] Unbalanced contribution framing.

The agent pipeline provides the outline and story history that help the critic locate the error. CEPO then uses that boundary to preserve positive credit for the preceding text and uses token entropy to weight the Writer’s update near the error.

As the title and the final paragraph of the Introduction indicate, our main contribution is CEPO. The agent pipeline provides the setting for testing it.

  • We introduce a critic-guided reinforcement learning framework for long-form generation agents, in which a Critic Agent provides localized feedback that directly optimizes the Writer Agent's parameters.
  • Through entropy analysis and a systematic comparison of sequence-level, span-level, and suffix-level strategies, we propose suffix-level credit assignment for long-form generation, preserving coherent prefixes while penalizing continuations conditioned on an identified error.
  • We develop an entropy-modulated token-level credit-assignment mechanism that dynamically scales optimization weights using predictive uncertainty, enabling targeted correction around narrative failures while preserving generation diversity.

[W3] Questionable assumptions in the reward design.

The first detected error provides a useful boundary for assigning credit, even when the story later recovers or subsequent errors arise independently.

Section 4.2 considers three penalty rules. The uniform penalty applies to the entire sequence, the span-level penalty applies only to flagged spans, and the suffix-level penalty preserves credit for the prefix and penalizes the continuation from the first flagged error onward. On Qwen3-8B, the suffix-level penalty reached the highest training critic score, followed by the span-level and uniform penalties (Appendix D, Figure 7).

Appendix F counts a trajectory as recovered if the critic detects no further error of the same type. Only 496 of 1,720 trajectories (28.8%) met this criterion (Table 5). Recovery was therefore relatively uncommon under this definition.

In addition, we consider two perturbations to the reward computation. The first moves the start of the suffix penalty from the first detected error to the second, testing how sensitive the token-level critic score is to missing the earliest flagged error.

Sensitivity to the error boundary. The first perturbation moves the start of the suffix penalty from the first detected error to the second, testing how the token-level critic score changes when the earliest flagged error is skipped.

Gradient Steps(0,80](80,160](160,240](240,320](320,400](400,480](480,560](560,640](640,720](720,800]
First Detected Error (CEPO)0.862 ± 0.0050.866 ± 0.0020.882 ± 0.0110.879 ± 0.0080.893 ± 0.0100.907 ± 0.0070.903 ± 0.0080.909 ± 0.0090.911 ± 0.0050.913 ± 0.003
Second Detected Error0.861 ± 0.0060.865 ± 0.0050.874 ± 0.0100.877 ± 0.0090.883 ± 0.0110.891 ± 0.0090.887 ± 0.0120.894 ± 0.0080.892 ± 0.0100.896 ± 0.007

Sensitivity to the training critic. The second perturbation retains the first-error rule and replaces Gemini-2.5-Flash-Lite with Gemini-2.5-Flash as the training critic.

Gradient Steps(0,80](80,160](160,240](240,320](320,400](400,480](480,560](560,640](640,720](720,800]
Gemini-2.5-Flash-Lite0.862 ± 0.0050.866 ± 0.0020.882 ± 0.0110.879 ± 0.0080.893 ± 0.0100.907 ± 0.0070.903 ± 0.0080.909 ± 0.0090.911 ± 0.0050.913 ± 0.003
Gemini-2.5-Flash0.862 ± 0.0060.868 ± 0.0040.876 ± 0.0090.881 ± 0.0070.888 ± 0.0090.897 ± 0.0080.900 ± 0.0070.897 ± 0.0100.904 ± 0.0060.905 ± 0.005

Both variants finished above their first-window means. Over the final 80 gradient steps, the mean critic score was 0.896 when the first error was skipped and 0.905 when Gemini-2.5-Flash replaced Gemini-2.5-Flash-Lite, compared with 0.913 for the shared baseline.

[W4] Overclaiming the granularity and causal interpretation.

CEPO uses a suffix-level reward and computes a separate entropy weight for each token. On Qwen3-8B, the suffix-level penalty achieved a higher training critic score than the span-level penalty (Appendix D, Figure 7). Appendix E finds lower entropy before flagged errors and higher entropy afterward relative to non-error positions matched on entropy at the onset. Equation (3) scales each token’s advantage by its weight. Equation (4) discounts low-entropy tokens before the error, upweights the onset, and increases the penalty on high-entropy tokens afterward, with the adjustment decaying away from the onset. Here, causal refers to the autoregressive dependence of each later token on the preceding text, including any earlier error.


[W5] Narrow target problem and incomplete experimental correspondence.

In the same-backbone comparisons in Table 1, we use the same generation pipeline, keep the Idea and Blueprint Agents fixed, and vary only the reinforcement learning method used to train the Writer Agent. On Qwen3-32B, CEPO scores higher than DAPO on all six evaluation dimensions, with an overall mean of 4.03 versus 3.71. The gains extend beyond coherence (3.25 vs. 3.11) to dimensions such as pacing (4.30 vs. 3.74) and style (3.63 vs. 3.31). These differences therefore reflect the choice of Writer training method within a fixed pipeline.

We designed our evaluation to assess novel quality along six dimensions: pacing, appeal, completeness, coherence, style, and hook. To examine the contribution of each part of CEPO, we conducted three ablation studies on Qwen3-8B. First, we compared uniform, span-level, and suffix-level penalties. The suffix-level penalty achieved the highest training critic score (Appendix D, Figure 7). Second, we kept the suffix-level reward fixed and compared the full modulation scheme with no modulation and with each of its three weighting components enabled individually (Section 5.4, Figure 4b). Third, we used a 2 × 2 ablation to vary entropy weighting and temporal decay independently while keeping the weight at the error onset fixed (Appendix G, Figure 13).

Thank you for the response. I appreciate the authors taking the time to clarify their intended contributions and to point me to the additional analyses.

  • W1–W2 (Writing and contribution framing): I understand that the authors consider CEPO the main contribution and the multi-agent pipeline its setting. The response clarifies this and restates the contributions in a helpful list. My concern, though, is about how the paper itself reads: the Introduction moves quickly between the pipeline, RL training, and sparse rewards without making the connections clear, and the pipeline still appears more central than the authors intend. If the authors prefer the current presentation, I respect that choice, but I maintain this concern.

  • W3 (Reward design): The penalty comparisons, recovery analysis, and boundary sensitivity experiment give me enough reason to accept the first-error boundary as a practical training choice. There are still cases where the critic may place the boundary incorrectly or the story may recover afterward, so I would view the suffix rule as a useful heuristic rather than a precise account of responsibility for every later token.

  • W4 (Granularity and “causal”): The explanation helped me understand the token-specific entropy weights, and I accept the authors’ intended meaning of “causal” as autoregressive dependence. Some of the wording in the paper still sounds stronger than that interpretation, but I no longer consider this a central concern.

  • W5 (Interpreting the gains): The fixed-pipeline comparison makes it much clearer that CEPO contributes to the reported gains, including gains beyond coherence. What I still cannot quite explain is why a reward centered on structural errors leads to improvements in pacing and style. A more direct analysis connecting the errors CEPO reduces to those evaluation dimensions would help.

Reviewer_GRDyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Causal Entropy Policy Optimization for long-form narrative generation. The method uses an LLM-based critic to identify the first structural error in a generated narrative and assigns negative rewards to the suffix following that error, while using token-level entropy to modulate the reward signal. The motivation is to improve credit assignment in long-form generation, where an early mistake may propagate and affect the quality of subsequent text.

The paper evaluates CEPO using several models, with comparisons to existing policy optimization approaches and alternative reward assignment strategies. It also examines the relationship between token entropy and narrative errors, evaluates the critic against human annotations, and reports both automated and human evaluations of generated novels. The results suggest that CEPO can improve the reported narrative-quality scores and that suffix-level reward assignment performs favorably among the tested alternatives.

Critical strengths
  1. An important problem with a plausible solution direction. Long-form narrative generation poses a challenging credit-assignment problem because errors early in a text can influence many subsequent tokens. Assigning rewards based on the location of the first detected error is a reasonable attempt to provide more targeted supervision than sequence-level rewards.

  2. A relatively broad empirical investigation. The paper evaluates two model scales, compares alternative reward assignment strategies, examines entropy patterns around detected errors, and includes human evaluation. The attempt to connect a token-level signal to narrative-level quality is interesting and provides a useful starting point for further investigation.

Critical weaknesses
  1. The reliability of the critic is a significant source of uncertainty. The manual audit reports materially different precision-recall trade-offs across critic models. For the selected Gemini-2.5-Flash-Lite critic, the reported precision is 79.53% and recall is 64.10% on a sample of 200 trajectories. Thus, a nontrivial fraction of predicted errors may be false positives, while many actual errors may remain undetected. Since the location of the first detected error determines the reward assigned to a potentially long suffix, critic mistakes can affect a large number of token-level training signals. The paper does not sufficiently establish how robust the resulting optimization is to this noise.

  2. The independent contribution of entropy modulation is not sufficiently isolated. The observed association between entropy patterns and narrative errors is suggestive, but it does not by itself establish that entropy provides a useful causal signal for improving policy optimization. The comparison of reward assignment strategies helps motivate suffix-level penalties, but the evidence does not clearly separate the effect of entropy modulation from the effect of suffix-level reward assignment and other training choices. A controlled ablation that holds the critic, reward assignment, training budget, and other relevant settings fixed while varying only entropy modulation would be important for validating the central claim. The paper would also benefit from clearer uncertainty estimates and statistical testing for the principal comparisons.

Questions for authors
  1. Entropy contribution: Can the authors provide a controlled ablation comparing CEPO with an otherwise identical method that uses the same critic and suffix-level reward assignment but removes entropy modulation? Were the comparisons conducted with matched training budgets and multiple random seeds, and are the gains statistically reliable?

  2. Critic errors and reward robustness: How sensitive are the results to false-positive and false-negative critic predictions? Can the authors quantify how critic errors affect the first-error position and the resulting suffix-level rewards, or provide a robustness experiment using a more accurate critic or controlled perturbations of the error labels?

  3. Statistical support for quality improvements: For the main comparisons, what are the numbers of independently evaluated novels, the aggregation units, and the uncertainty estimates? In particular, can the authors clarify whether statistical tests account for multiple judgments of the same novel or other dependencies in the human and LLM evaluations?

  4. Dataset provenance: The paper describes a dataset of 81,496 web novels with full copyright ownership. Can the authors clarify how copyright ownership or licensing was established, and whether the data can be used for model training and the reported evaluation under the applicable terms?

1 comment

[W1][Q2] Critic errors and reward robustness.

We consider two perturbations to the reward computation. The first moves the start of the suffix penalty from the first detected error to the second, testing how sensitive the token-level critic score is to missing the earliest flagged error.

Gradient Steps(0,80](80,160](160,240](240,320](320,400](400,480](480,560](560,640](640,720](720,800]
First Detected Error (CEPO)0.862 ± 0.0050.866 ± 0.0020.882 ± 0.0110.879 ± 0.0080.893 ± 0.0100.907 ± 0.0070.903 ± 0.0080.909 ± 0.0090.911 ± 0.0050.913 ± 0.003
Second Detected Error0.861 ± 0.0060.865 ± 0.0050.874 ± 0.0100.877 ± 0.0090.883 ± 0.0110.891 ± 0.0090.887 ± 0.0120.894 ± 0.0080.892 ± 0.0100.896 ± 0.007

The second perturbation retains the first-error rule and replaces Gemini-2.5-Flash-Lite with Gemini-2.5-Flash as the training critic.

Gradient Steps(0,80](80,160](160,240](240,320](320,400](400,480](480,560](560,640](640,720](720,800]
Gemini-2.5-Flash-Lite0.862 ± 0.0050.866 ± 0.0020.882 ± 0.0110.879 ± 0.0080.893 ± 0.0100.907 ± 0.0070.903 ± 0.0080.909 ± 0.0090.911 ± 0.0050.913 ± 0.003
Gemini-2.5-Flash0.862 ± 0.0060.868 ± 0.0040.876 ± 0.0090.881 ± 0.0070.888 ± 0.0090.897 ± 0.0080.900 ± 0.0070.897 ± 0.0100.904 ± 0.0060.905 ± 0.005

Both variants finished above their first-window means. Over the final 80 gradient steps, the mean critic score was 0.896 when the first error was skipped and 0.905 when Gemini-2.5-Flash replaced Gemini-2.5-Flash-Lite, compared with 0.913 for the shared baseline.


[W2][Q1] What does the ablation show about the contribution of entropy modulation?

As described in Section 5.4, CEPO uses three modulation parameters, α\alpha, β\beta, and γ\gamma. In Figure 4b, None sets (α,β,γ)=(0,1,0)(\alpha,\beta,\gamma)=(0,1,0), so every token has a modulation weight of one. This configuration retains CEPO’s suffix-level reward without applying entropy modulation. All sets (α,β,γ)=(1,2,1)(\alpha,\beta,\gamma)=(1,2,1), enabling the full CEPO weighting scheme. Comparing All with None therefore measures the effect of enabling the full modulation scheme while holding the suffix-level reward fixed. The table below reports the mean token-level critic score for each configuration over successive windows of 80 gradient steps.

Gradient steps(0,80](80,160](160,240](240,320](320,400](400,480](480,560](560,640](640,720](720,800]
Without entropy modulation (None)0.866 ± 0.0100.869 ± 0.0060.882 ± 0.0080.873 ± 0.0130.889 ± 0.0110.895 ± 0.0080.889 ± 0.0120.899 ± 0.0110.892 ± 0.0110.896 ± 0.009
With entropy modulation (All)0.862 ± 0.0050.866 ± 0.0020.882 ± 0.0110.879 ± 0.0080.893 ± 0.0100.907 ± 0.0070.903 ± 0.0080.909 ± 0.0090.911 ± 0.0050.913 ± 0.003

[Q3] What is the sample size, and do the tests account for repeated judgments?

The test set contains 100 held-out story prompts. For each generated novel and each evaluation dimension, three judges (Gemini-2.5-Flash, DeepSeek-R1, and GLM-5.1) each scored the same output five times. This yields 15 ratings per novel per dimension.

The scores {1,3,5}\{1,3,5\} represent ordered categories. The standard deviation is not strictly appropriate for summarizing variation on this ordinal scale because the numerical gaps between scores need not represent equal differences in quality. In Appendix H, we use two-sided Mann–Whitney U tests to compare the rating distributions of Qwen3-32B CEPO and Gemini-2.5-Flash on each dimension. These rank-based tests do not assume equal distances between adjacent rating categories.

The overall mean scores were 4.03 for Qwen3-32B CEPO and 3.81 for Gemini-2.5-Flash. On the same Qwen3-32B backbone, CEPO also scored 0.32 points above DAPO (4.03 vs. 3.71).

For the human evaluation, ten readers compared complete novels in pairs. Model identities were hidden, and presentation order was randomized. Inter-rater agreement, measured by Fleiss’ κ\kappa, was 0.61 for pacing, 0.54 for appeal, 0.72 for completeness, 0.68 for coherence, 0.51 for style, and 0.64 for hook.


[Q4] Where did the novels come from, and what rights cover their use?

The 81,496 novels in our dataset come from the web-fiction platform we operate. When users upload their novels, they can choose whether to grant the platform permission to use those works for research purposes.

Reviewer_FGN4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies long-form text generation and proposes an agentic pipeline that decomposes the generation process into intermediate steps. It also introduces CEPO, an RL method that uses an LLM critic to identify the first narrative error, assigns suffix-level penalties, and modulates token-level advantages around the error position based on entropy. Experiments on novels collected from the web show that the proposed method outperforms the baselines in both automatic and human evaluations.

Critical strengths
  • The paper proposes an agentic pipeline to decompose long-form text generation into intermediate steps, and proposes a novel RL method for credit-assignment;
  • Results with both automatics and human evaluations on multiple aspects show that CEPO improves several narrative-quality dimensions.
Critical weaknesses
  • Several technical details require clarification. Equation 2 defines the token-level reward, while Equation 3 introduces a modulated advantage based on a “raw reward.” Do these refer to the same reward? How are token-level advantages computed, are they group normalization as in GRPO? Does it contain any returns-to-go or baseline? Furthermore, given the agentic generation pipeline, are errors identified only in the final story, or can they also be detected in intermediate stages?
  • The reliability of the critic needs further investigation. The method relies exclusively on Gemini-2.5 for error identification, but the critic itself may make mistakes, particularly when evaluating extremely long outputs. How is the accuracy of its error localization validated, and what mechanisms mitigate potential reward hacking? In addition, RL training uses only 1,000 samples. This limited training scale makes it difficult to assess whether CEPO’s advantages over other RL baselines hold more broadly.
  • The novelty appears somewhat limited. Decomposing long-form text generation into an agentic pipeline is not new, and the paper’s main contribution appears to be CEPO. This narrows the scope of the overall contribution.
  • The paper lacks detailed qualitative examples and analyses of representative successes and failures. Key statistics, such as generated story lengths and token usage across pipeline stages, are also missing, making it difficult to assess the method’s effectiveness.
Questions for authors

See weaknesses

Also, what topics are these novels focused? Is that possible the generated articles may contain factual errors or hallucinations, especially for some certain topics? This should be better discussed in the data section.

4 comments

[W1] How do Eqs. (2) and (3) relate, how are token-level advantages computed, and which stages receive error feedback?

Reward and advantage. Equation (2) defines the immediate token reward ri,tr_{i,t}. For a failed Writer rollout ii, the Critic identifies the first error at position ti∗t_i^{\ast}. Tokens before ti∗t_i^{\ast} receive +1+1, and tokens from ti∗t_i^{\ast} onward receive −1-1. The “raw” quantity in Eq. (3) is the token advantage Ai,tA_{i,t} computed from this reward before entropy modulation. Both equations thus use the same underlying reward signal.

Group normalization and baseline. For each prompt qq, we sample eight Writer rollouts. A scored rollout receives the label yi=−1y_i=-1 if any valid response token has a negative reward, and yi=+1y_i=+1 otherwise. Let μq\mu_q and σq\sigma_q denote the mean and population standard deviation of the scored rollout labels for that prompt. For a group containing both labels, the raw and modulated advantages are

Ai,t=ri,t∣yi−μqσq+ε∣,A^i,tCEPO=ωi,tAi,t.A_{i,t} = r_{i,t}\left|\frac{y_i-\mu_q}{\sigma_q+\varepsilon}\right|, \qquad \widehat A_{i,t}^{\mathrm{CEPO}} = \omega_{i,t}A_{i,t}.

As in GRPO, the prompt-group mean serves as the baseline. CEPO uses the magnitude of the normalized rollout label as a scale shared by all tokens in that rollout. Each token’s immediate reward determines the sign of its raw advantage, so a failed rollout can provide positive credit for its coherent prefix and negative credit from the error onset onward. This computation uses neither returns-to-go nor a learned value-function baseline. Equation (3) then applies the token-specific weight ωi,t\omega_{i,t}, whose adjustment decays with distance from the error onset. We whiten the modulated advantages over valid response tokens within each mini-batch before the clipped policy update.

Feedback across generation stages. Only errors localized to Writer-generated chapter blocks are included in the RL objective. We focus on the Writer Agent because it produces the novel that readers evaluate, and keeping the Idea Agent and the Blueprint Agent fixed lets us study changes in Writer Agent behavior under the same narrative constraints.

[W2] The reliability of the critic needs further investigation, and RL training uses only 1,000 samples.

Error localization. In Appendix B, human annotators audited 200 trajectories. They checked whether each flagged span contained a genuine error and inspected the underlying text for errors the critic missed. Against this human assessment, the training critic, Gemini-2.5-Flash-Lite, achieved 79.53% precision and 64.10% recall. Gemini-2.5-Flash achieved 96.95% precision and 19.10% recall on the same audit.

Sensitivity to the reward rule. We consider two perturbations to the reward computation. The first moves the start of the suffix penalty from the first detected error to the second, testing how sensitive the token-level critic score is to missing the earliest flagged error.

Gradient Steps(0,80](80,160](160,240](240,320](320,400](400,480](480,560](560,640](640,720](720,800]
First Detected Error (CEPO)0.862 ± 0.0050.866 ± 0.0020.882 ± 0.0110.879 ± 0.0080.893 ± 0.0100.907 ± 0.0070.903 ± 0.0080.909 ± 0.0090.911 ± 0.0050.913 ± 0.003
Second Detected Error0.861 ± 0.0060.865 ± 0.0050.874 ± 0.0100.877 ± 0.0090.883 ± 0.0110.891 ± 0.0090.887 ± 0.0120.894 ± 0.0080.892 ± 0.0100.896 ± 0.007

The second perturbation retains the first-error rule and replaces Gemini-2.5-Flash-Lite with Gemini-2.5-Flash as the training critic.

Gradient Steps(0,80](80,160](160,240](240,320](320,400](400,480](480,560](560,640](640,720](720,800]
Gemini-2.5-Flash-Lite0.862 ± 0.0050.866 ± 0.0020.882 ± 0.0110.879 ± 0.0080.893 ± 0.0100.907 ± 0.0070.903 ± 0.0080.909 ± 0.0090.911 ± 0.0050.913 ± 0.003
Gemini-2.5-Flash0.862 ± 0.0060.868 ± 0.0040.876 ± 0.0090.881 ± 0.0070.888 ± 0.0090.897 ± 0.0080.900 ± 0.0070.897 ± 0.0100.904 ± 0.0060.905 ± 0.005

Both variants finished above their first-window means. Over the final 80 gradient steps, the mean critic score was 0.896 when the first error was skipped and 0.905 when Gemini-2.5-Flash replaced Gemini-2.5-Flash-Lite, compared with 0.913 for the shared baseline.

RL data scale. In long-form writing, Writing-RL selected 1,500 prompts from each of two datasets for RL training (Lei et al., 2026). It is relatively reasonable that our RL split contains 1,000 examples.

[1] Xuanyu Lei, Chenliang Li, Yuning Wu, Kaiming Liu, Weizhou Shen, Peng Li, Ming Yan, Fei Huang, Ya-Qin Zhang, and Yang Liu. Writing-RL: Advancing long-form writing via adaptive curriculum reinforcement learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5639–5661, 2026.


[W3] What is novel beyond the agentic writing pipeline?

As the title and the final paragraph of the Introduction indicate, our main contribution is CEPO.

  • We introduce a critic-guided reinforcement learning framework for long-form generation agents, in which a Critic Agent provides localized feedback that directly optimizes the Writer Agent's parameters.
  • Through entropy analysis and a systematic comparison of sequence-level, span-level, and suffix-level strategies, we propose suffix-level credit assignment for long-form generation, preserving coherent prefixes while penalizing continuations conditioned on an identified error.
  • We develop an entropy-modulated token-level credit-assignment mechanism that dynamically scales optimization weights using predictive uncertainty, enabling targeted correction around narrative failures while preserving generation diversity.

[Q1] Which topics do the novels cover, and can the generated stories contain factual errors?

As described in Appendix A, our dataset contains 81,496 web novels across 12 genres, including workplace drama, suspense and horror, folklore and legends, and post-apocalyptic science fiction. Generated stories can contain factual errors even when they follow the outline. For example, a novel may give the wrong date for a historical event and repeat that date throughout later chapters. When scoring coherence, we consider both the narrative’s internal consistency and factual errors of this kind.

[W4] How many tokens does generation use?

Table 1 reports mean final novel-body lengths of approximately 33k tokens for Qwen3-8B CEPO and 34k for Qwen3-32B CEPO. The table below gives mean token counts per block.

Recorded textTokens
Story setting / chapter outline / previous content1,426 / 868 / 3,083
Generated chapter text2,310
Complete critic prompt / critic response8,940 / 412

[W4] What do representative successes and failures look like?

As shown in Appendix M, the relevant logical error cases are as follows:

Note: The tokens wrapped in angle brackets in both the English translations and the original Chinese texts represent the zero-point tokens.

 

Entity Consistency

Entity Definition:

墨景深:权势滔天的豪门掌权者,深情隐忍,两世守护女主,是女主复仇路上的最强后盾。

(Mo Jingshen: A powerful mastermind of a wealthy family, deeply affectionate and forbearing, guarding the female lead through two lifetimes, and serving as her strongest backer on her path to revenge.)

Model Generated Text:

我捂着嘴,眼泪止不住地往下掉。我终于明白了,<原>来我所有的磨难,都是他计划的一部分。他是一直在默默守护我的那个人。

(I covered my mouth, tears falling uncontrollably. I finally understood that, <turns> out all my suffering was part of his plan. He was the one who had been silently guarding me all along.)

Reason for Bad Case:

墨景深的人设是“深情隐忍,两世守护女主”,此句“所有的磨难,都是他计划的一部分”与墨景深作为守护者的角色完全矛盾。

(Mo Jingshen's character setting is "deeply affectionate and forbearing, guarding the female lead through two lifetimes." The sentence "all my suffering was part of his plan" completely contradicts Mo Jingshen's role as a guardian.)

 

Outline Alignment

Outline:

多年后,陆彦辰在边远地区经过一番磨砺,逐渐成熟。墨景深给予了他重新开始的机会,让他能够凭借自己的努力,在新的领域闯出一片天地。

(Years later, Lu Yanchen gradually matures after honing himself in a remote area. Mo Jingshen gives him an opportunity to start over, allowing him to carve out a place in a new field through his own efforts.)

Model Generated Text:

他告诉墨景深,陆彦辰已经在那个偏远小镇站稳了脚跟,开了个小公司,虽然赚不了大钱,但人总算踏实了。老爷子叹了口气,<说>陆家没有抛弃他,给了他重新开始的机会。墨景深说,那也是他应得的。我听到这里,心里很平静。那些人,那些事,对我来说,真的已经过去了。

(He told Mo Jingshen that Lu Yanchen had already gained a firm foothold in that remote small town and opened a small company. Although he couldn't make big money, he was finally settled. The old man sighed and <said> the Lu family had not abandoned him, giving him a chance to start over. Mo Jingshen said that it was what he deserved. Hearing this, I felt very peaceful. Those people and those things are truly in the past for me.)

Reason for Bad Case:

故事大纲中明确指出是“墨景深给予了他(陆彦辰)重新开始的机会”,而正文描述为“陆家没有抛弃他,给了他重新开始的机会”。

(The story outline explicitly states that "Mo Jingshen gives him (Lu Yanchen) an opportunity to start over," while the main text describes it as "the Lu family had not abandoned him, giving him a chance to start over.")

 

Hard Constraints

Entity Definition:

苏辰:第一人称主角,为父复仇,隐忍狠绝,以屈辱为伪装的天才鉴定师,代号“画师”。

(Su Chen: First-person protagonist, seeking revenge for his father, forbearing and ruthless, a genius appraiser who uses humiliation as a disguise, codenamed "Painter.")

Model Generated Text:

天穹阁的声誉开始急剧下滑,内部也开始出现恐慌。秦晚试图动用公关力量压制,但<苏>辰巧妙地利用网络匿名性,让信息如同野火般蔓延,难以扑灭。我站在高楼的窗前,看着城市灯火,心中充满了复杂的情绪。

(Tianqiong Pavilion's reputation began to decline sharply, and panic began to emerge internally. Qin Wan tried to use public relations to suppress it, but <Su> Chen cleverly used internet anonymity to make the information spread like wildfire, making it difficult to extinguish. I stood in front of the window of the tall building, looking at the city lights, my heart filled with complex emotions.)

Reason for Bad Case:

文本中出现人称转换,从第一人称“我”突然变为第三人称“苏辰”,严重违反小说第一人称设定。

(A point-of-view shift occurs in the text, suddenly changing from the first-person "I" to the third-person "Su Chen," which severely violates the first-person setting of the novel.)

 

Spatiotemporal Logic

Model Generated Text:

就在这时,我身后的大屏幕亮起,不再是设计图,而是一段录音。「你放心,那些账目,我已经做平了。等公司上头,钱就是我们的了。」是沈月的声音。<「>三年前那场火,也是你安排的?」我追问道。

(Just then, the large screen behind me lit up, no longer showing the design drafts, but playing an audio recording. "Don't worry, I have already balanced those accounts. Once the company goes public, the money will be ours." It was Shen Yue's voice. <「> Was the fire three years ago also arranged by you?」 I pressed on.)

Reason for Bad Case:

旁白中插入了疑问“三年前那场火,也是你安排的?”,并紧接着“我追问道”。但是录音是预先存在的对话,现时提问不可能出现在录音内容中。

(A question ("Was the fire three years ago also arranged by you?") is inserted into the narration, immediately followed by "I pressed on." However, the recording is a pre-existing conversation, and the present-time question cannot possibly appear within the recorded content.)

Thanks for the authors' response, which has addressed some of my questions and concerns. I tend to maintain my overall rating.

Comments on the paper

2 comments
Area_Chair_jenhOpenReview ↗

Reminder: Discussion deadline approaching

Dear Reviewers,

Thank you for your time and effort in reviewing Submission 23366. As the discussion deadline is approaching, could you please read the authors’ response and any additional evidence, reply in the discussion to indicate whether your concerns have been addressed, and update your evaluation accordingly as soon as possible? Thank you for your contribution to the review process.

Best regards, Area Chair

We thank all reviewers for recognizing the value of our work on credit assignment for long-form narrative generation and the improvements demonstrated in our evaluations. We also appreciate their suggestions on aspects such as the paper’s presentation, critic reliability, and statistical support for the reported improvements.

We added two sensitivity experiments: shifting the penalty onset to the second detected error and replacing the training critic, examining sensitivity to missed errors and critic choice. We also clarified the reward-design and entropy-modulation ablations and discussed the human audit of critic predictions.

Our responses further explain how token rewards are converted into group-relative advantages and provide details on the evaluation sample size, repeated scoring procedure, statistical tests, and blinded human evaluation, including inter-rater agreement. We also revised the Introduction to clarify the motivation.

CEPO combines suffix-level supervision with entropy-guided token weighting. It preserves positive credit before the first detected error, penalizes the continuation from that error onward, and uses entropy-guided weighting to adjust token-level update strength around the error, providing a novel and practical approach to credit assignment for long-form writing. We hope these additions and revisions clarify CEPO’s contribution and address the reviewers’ main concerns. We have revised the manuscript, with all changes marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.