- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a hierarchical multi-agent framework for long-form novel generation, combining planning, memory, world modeling, and critique. It further proposes CEPO, which uses an LLM critic to locate the first structural error and assigns suffix-level, entropy-modulated rewards to train the Writer Agent. Experiments on web-novel data show improvements over several RL baselines in automated and human evaluations.
Critical strengths
The paper addresses a practically relevant challenge in long-form generation, namely sparse and delayed feedback caused by error propagation over long contexts. The proposed reward design is relatively simple and interpretable, and can potentially be integrated into existing training pipelines. In addition, the empirical analysis before the method provides some motivation for the proposed error-localized reward scheme.
Critical weaknesses
-
Unclear motivation and narrative. The introduction presents the multi-agent framework, RL training, and sparse-reward problem almost simultaneously, without establishing a clear logical chain explaining why each component is necessary. As a result, the paper appears to introduce a complex system before clearly defining the underlying problem.
-
Unbalanced contribution framing. The multi-agent framework is largely an engineering scaffold, and its individual components do not appear particularly novel. However, the paper presents the framework and CEPO as comparably important contributions, which may lead readers to overestimate the novelty of the overall system. More controlled comparisons are needed to isolate the contribution of CEPO.
-
Questionable assumptions in the reward design. CEPO relies on an LLM critic to identify the first error and assumes that subsequent tokens are primarily affected by this error. These assumptions may be unreliable, since the critic can misidentify the error location, later content may recover from an earlier mistake, and multiple independent errors may occur in the same sequence.
-
Overclaiming the granularity and causal interpretation. The proposed reward remains a suffix-level signal determined by a single error boundary, rather than genuinely fine-grained token-level credit assignment. Moreover, terms such as “causal” appear stronger than what is supported by the method, since the paper does not establish a causal relationship between the identified error and all subsequent tokens. The entropy-based weighting also seems largely heuristic.
-
Narrow target problem and incomplete experimental correspondence. The method focuses mainly on consistency degradation after an early error, while novel generation involves many other dimensions such as plot quality, characterization, style, and readability. In addition, the experiments do not fully isolate or directly validate the proposed mechanism, making it difficult to determine whether the reported gains are specifically due to CEPO or to other components of the framework.
Questions for authors
please see weakness above
5 comments
[W4] Overclaiming the granularity and causal interpretation.
CEPO uses a suffix-level reward and computes a separate entropy weight for each token. On Qwen3-8B, the suffix-level penalty achieved a higher training critic score than the span-level penalty (Appendix D, Figure 7). Appendix E finds lower entropy before flagged errors and higher entropy afterward relative to non-error positions matched on entropy at the onset. Equation (3) scales each token’s advantage by its weight. Equation (4) discounts low-entropy tokens before the error, upweights the onset, and increases the penalty on high-entropy tokens afterward, with the adjustment decaying away from the onset. Here, causal refers to the autoregressive dependence of each later token on the preceding text, including any earlier error.
[W5] Narrow target problem and incomplete experimental correspondence.
In the same-backbone comparisons in Table 1, we use the same generation pipeline, keep the Idea and Blueprint Agents fixed, and vary only the reinforcement learning method used to train the Writer Agent. On Qwen3-32B, CEPO scores higher than DAPO on all six evaluation dimensions, with an overall mean of 4.03 versus 3.71. The gains extend beyond coherence (3.25 vs. 3.11) to dimensions such as pacing (4.30 vs. 3.74) and style (3.63 vs. 3.31). These differences therefore reflect the choice of Writer training method within a fixed pipeline.
We designed our evaluation to assess novel quality along six dimensions: pacing, appeal, completeness, coherence, style, and hook. To examine the contribution of each part of CEPO, we conducted three ablation studies on Qwen3-8B. First, we compared uniform, span-level, and suffix-level penalties. The suffix-level penalty achieved the highest training critic score (Appendix D, Figure 7). Second, we kept the suffix-level reward fixed and compared the full modulation scheme with no modulation and with each of its three weighting components enabled individually (Section 5.4, Figure 4b). Third, we used a 2 × 2 ablation to vary entropy weighting and temporal decay independently while keeping the weight at the error onset fixed (Appendix G, Figure 13).
Thank you for the response. I appreciate the authors taking the time to clarify their intended contributions and to point me to the additional analyses.
-
W1–W2 (Writing and contribution framing): I understand that the authors consider CEPO the main contribution and the multi-agent pipeline its setting. The response clarifies this and restates the contributions in a helpful list. My concern, though, is about how the paper itself reads: the Introduction moves quickly between the pipeline, RL training, and sparse rewards without making the connections clear, and the pipeline still appears more central than the authors intend. If the authors prefer the current presentation, I respect that choice, but I maintain this concern.
-
W3 (Reward design): The penalty comparisons, recovery analysis, and boundary sensitivity experiment give me enough reason to accept the first-error boundary as a practical training choice. There are still cases where the critic may place the boundary incorrectly or the story may recover afterward, so I would view the suffix rule as a useful heuristic rather than a precise account of responsibility for every later token.
-
W4 (Granularity and “causal”): The explanation helped me understand the token-specific entropy weights, and I accept the authors’ intended meaning of “causal” as autoregressive dependence. Some of the wording in the paper still sounds stronger than that interpretation, but I no longer consider this a central concern.
-
W5 (Interpreting the gains): The fixed-pipeline comparison makes it much clearer that CEPO contributes to the reported gains, including gains beyond coherence. What I still cannot quite explain is why a reward centered on structural errors leads to improvements in pacing and style. A more direct analysis connecting the errors CEPO reduces to those evaluation dimensions would help.
[W1][W2] Unbalanced contribution framing.
The agent pipeline provides the outline and story history that help the critic locate the error. CEPO then uses that boundary to preserve positive credit for the preceding text and uses token entropy to weight the Writer’s update near the error.
As the title and the final paragraph of the Introduction indicate, our main contribution is CEPO. The agent pipeline provides the setting for testing it.
[W3] Questionable assumptions in the reward design.
The first detected error provides a useful boundary for assigning credit, even when the story later recovers or subsequent errors arise independently.
Section 4.2 considers three penalty rules. The uniform penalty applies to the entire sequence, the span-level penalty applies only to flagged spans, and the suffix-level penalty preserves credit for the prefix and penalizes the continuation from the first flagged error onward. On Qwen3-8B, the suffix-level penalty reached the highest training critic score, followed by the span-level and uniform penalties (Appendix D, Figure 7).
Appendix F counts a trajectory as recovered if the critic detects no further error of the same type. Only 496 of 1,720 trajectories (28.8%) met this criterion (Table 5). Recovery was therefore relatively uncommon under this definition.
In addition, we consider two perturbations to the reward computation. The first moves the start of the suffix penalty from the first detected error to the second, testing how sensitive the token-level critic score is to missing the earliest flagged error.
Sensitivity to the error boundary. The first perturbation moves the start of the suffix penalty from the first detected error to the second, testing how the token-level critic score changes when the earliest flagged error is skipped.
Sensitivity to the training critic. The second perturbation retains the first-error rule and replaces Gemini-2.5-Flash-Lite with Gemini-2.5-Flash as the training critic.
Both variants finished above their first-window means. Over the final 80 gradient steps, the mean critic score was 0.896 when the first error was skipped and 0.905 when Gemini-2.5-Flash replaced Gemini-2.5-Flash-Lite, compared with 0.913 for the shared baseline.