- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A Framework Paper on Agent Self-Evolution. By using the diffusion model "corruption-reconstruction" and text gradient back propagation, the 20 professional short plays were pressed into an outline, and the script agent reconstructed, the loss agent produced the semantic difference report, and the backward agent performed the addition, modification and merging of the text skill library organized by MoE, and the parameters were frozen in the whole process. 1,200 token linter hard regular, 6 novels 385 sets set, 10 event-level metrics + 5 people blind rating.
Critical strengths
1.The "professional finished product does not contain amateur errors" is transformed into an operational error-driven signal, that is, a self-supervised pair is constructed by corrosion-reconstruction, which is an exact design and is supported by +Distill control experiments.
2.Rule linter forces the abstraction of skill cards with hard capacity constraints, which is a practical engineering technique to curb the expansion of skill library, and brings a certain degree of attribution to the training process.
3.The phenomenon of "weak model mining rules, strong model execution" in cross-model experiments is a valuable side observation.
Critical weaknesses
Reproduction Concern. 1.The evaluation system is completely private and closed loop: neither the training corpus nor the reserved test set is distributed, the ten indicators are customized for this paper and there is no public benchmark anchor, two of the five readers are the authors, so all the core figures cannot be reproduced by the third party or compared horizontally.
2.The paper abandoned the public evaluation method on the basis of "LLM referee scoring is unreliable ", but its own five episode level indicators still rely on LLM referee, and it has never done meta-verification on the form of referee on public data, and there is no evidence whether the criticized failure mode has been eliminated.
3.Several key designs have only qualitative support: the contribution of key component without performance ablation.
Questions for authors
1.Could you run one inference-only control: give Opus the +Distill rule library (or any comparably strict, non-error-driven rule set) and compare it with Sonnet's loop-trained library? If the non-error-driven library matches the gains, the paper's central "error-driven" attribution is falsified — the benefit would come from rule strictness, not from the reconstruction loop.
2.Could you report, from your existing logs, the recurrence rate of errors after the corresponding rule was added to the library? If a large fraction of loss items re-flag errors that already have a matching rule, the credit-assignment claim is falsified — the backward agent would be compensating for retrieval/execution failures that the framework never models, rather than learning new skills.
3.Could you retrain the loop with the learned corruption step replaced by a fixed, non-learned compressor (e.g., extractive scene headers)? If the learned skills and downstream metrics are statistically unchanged, the diffusion-style corruption is falsified as a mechanism — it serves only to construct pairs, and the analogy should be removed from the framing.
4 comments
Response to Reviewer FhQ7 (Part 2 of 2): Q2, Q3
Q2. What is the recurrence rate of errors after the corresponding rule was added?
First, once a concrete rule is added to the library, the corresponding error drops sharply and stays low. As shown in Figure 7 of Appendix G, unfilmable action lines fall by 70% within the first five backward micro-steps, from 1.31 to 0.39 per episode. Over the full training, missing hard-cut endings drop by half, from 66% of episodes for Base to 33% for the final library (Table 2).
Second, we sampled 276 of the 687 loss reports from the 20-novel run, comprising 1,758 diagnosis items across all training steps, and checked how each item relates to the library at that step. 29% of the items had no matching rule in the library yet. 55% fell under a topic card that the forward agent had read, and 16% fell under a topic card that was not retrieved. GPT-5.5 and Claude Opus 4.8 agreed on these proportions within 1.5 percentage points.
Crucially, the 55% that fall under a read card do not mean that the agent ignored the card. Our library organizes knowledge into 23 broad topic cards that are mostly built early in Epoch 1, when most of the library growth happens (Figure 6b), so later problems naturally fall under these same cards. Each card bundles many rules within 1,200 tokens, and an early rule is usually stated as a broad principle. Different novels and different stages of a drama, such as an opening setup versus a mid-story betrayal or climax, present new situations where the broad rule is not yet concrete enough to be applied reliably. When the backward agent updates a read card, it rewrites the broad rule into a more concrete procedure for these situations. This is consistent with Appendix F, where libraries trained on more novels turn broad instructions into more precise, context-aware procedures. Similarly, for the 16% where a card was not retrieved, the backward agent broadens the card's description so that it is triggered in the newly encountered scene types. Credit assignment therefore keeps expanding the coverage and precision of existing cards, rather than merely compensating for execution failures.
Q3. Would replacing the learned corruption step with a fixed, non-learned compressor leave the results unchanged?
We clarify a slight misunderstanding here: just as the forward noising process in standard diffusion models (Ho et al., 2020) is a fixed, non-learned operator and only the reverse denoising process is trained, our corruption step is already a fixed, non-learned compressor. Specifically, it is a frozen LLM call with a fixed prompt that strips dialogue and action details into a story outline, receiving zero updates during training. Only the skill library used in the reverse reconstruction step is trained. What matters for our diffusion-inspired loop is that the forward corruption strips away domain craft while preserving the story skeleton, forcing the reconstruction step to recover that craft and exposing capability gaps when compared against the human original. Replacing the prompt-based summarizer with another fixed compressor that strips the same craft details would serve the same functional role. We will add these clarifications to the revised manuscript, such as the Opus control with the +Distill library, the recurrence analysis, and the role of the corruption step in Section 3.1.
Thank you for the substantial rebuttal. Running the Opus +Distill control during the rebuttal period is exactly what I asked for, and I accept its result: rule strictness alone does not explain the gains. The stability checks and the new external readers also help.
Still, a few things keep me at a 2. First, in your own table the trained library beats the human reference on 4 of 6 metrics (cold open 83% vs 59% for humans, prop activation 77% vs 57%). If these metrics really track professional craft, the system should approach human values, not shoot well past them. To me this pattern suggests the loop may be optimizing the LLM-judged metrics themselves, and I would like to see this addressed head-on rather than with more gains on the same metrics. Second, the comparison still lives entirely inside the private evaluation loop, and the +Distill arm differs from yours not only in the learning signal but also in size (131 vs 23 cards), which muddies the attribution. Third, SWE-bench Pro is encouraging, but it is a different domain and the setup details are in another thread; the screenplay claims themselves remain unverifiable by third parties. The new human study compares against an empty library, which is a low bar.
On Q2, 71% of loss items fall under cards that already existed. Your concretization reading is plausible, but a split of that 55% into "same specific rule re-flagged" vs "genuinely new situation" would settle it. On Q3, thanks for the clarification; I had misread the corruption step as learned. I would still suggest softening the diffusion framing, since with both directions frozen it is mostly a metaphor.
I appreciate the work that went into this response. With the human-overshoot pattern unexplained and the in-domain numbers still private, I keep my score at 2, though this rebuttal has genuinely narrowed my concerns.
Thank you very much for taking the time to review our rebuttal and for acknowledging that the Opus +Distill control, stability checks, and expanded human study have genuinely narrowed your concerns. We briefly address your remaining points below:
-
Human-overshoot & whether the loop optimizes the LLM-judged metrics: We addressed this head-on in Response to Reviewer rePM (W1 & Q1) and Appendix J (Table 18) of the revised PDF. First, the loss agent never sees the 10 evaluation metrics or judge prompts during training (Section 3.3, Appendix A), so it cannot optimize the metrics directly. Second, in a direct episode-level fidelity test asking two independent judges which script is closer in craft to the held-out human original of the same episode (Appendix J, Table 18),
+Oursis judged closer to the human original thanBase(84%),+Few-shot(90%), and+Distill(95%), and episode-level Cohen's on prop activation rises from 0.35 to 0.55. Why do aggregate hook rates exceed the human average? Because human writers use hooks in ~60% of episodes and relax pacing on transition episodes, whereas our compact rules apply hooks consistently across every episode—which is why the professional screenwriter preferred+Ourseven over the human original in 9 of 15 pairs (Appendix H, Table 16), noting that the human originals occasionally failed to hold the ending hook. -
Library size (131 vs. 23 cards) & inference budget: Both
+Distilland+Ourswere trained on the exact same 20 novels for 174 micro-steps under the identical 1,200-token card limit. More importantly, at inference time the agent does not load the entire library into context: under our MoE routing mechanism and identical context budget (Section 4.1, Appendix A), the agent only opens the subset of cards relevant to the current outline. Furthermore, in our 5-novel multi-run experiments (Appendix E.1, Table 12), where both libraries are much smaller,+Oursstill consistently outperforms+Distillon every directional metric. -
Human study comparisons, SWE-bench Pro details, and diffusion framing: Our expanded 8-reader study (Appendix H, Tables 15–16) does not only compare against an empty library—it explicitly evaluates against Human Originals as well (narrowing the net preference gap to human scripts from +69.2% to +41.7%), and the full SWE-bench Pro setup and results are integrated into Appendix I (Table 17) of the revised PDF. Unlike our screenplay corpus, SWE-bench Pro is fully public. Its tasks, repositories, and unit tests are all open, so third parties can directly verify our results on this benchmark. Finally, we appreciate your suggestion on Q3 and will gladly soften the diffusion framing to make clear that it serves as a conceptual inspiration for self-supervised pair construction.
Response to Reviewer FhQ7 (Part 1 of 2): W1, W2, W3 & Q1
We thank the reviewer for the thorough reading and for highlighting our error-driven learning signal, the rule linter constraint, and our cross-model findings. Below we address each weakness and question in detail.
W1. The evaluation is private and cannot be reproduced by third parties.
Because our training and test screenplays are commercial scripts under copyright, we cannot redistribute the raw text. To ensure our methodology and evaluation are fully reproducible on any screenplay corpus, we will open-source the complete training framework, the deterministic line-level parsers, the exact definitions of all ten metrics, and all LLM judge prompts. More importantly, to provide a fully public benchmark anchor with objective horizontal comparison, we also evaluated our exact corruption-and-reconstruction loop on the public SWE-bench Pro benchmark (Deng et al., 2025), where correctness is strictly verified by executable unit tests (
pass@1). On SWE-bench Pro with Claude Sonnet 4.6, our loop raisespass@1from 34.0% (Base) to 42.5%* (+Ours), also outperforming+Few-shot(35.5%) and+Distill(31.0%). Please see our response to Reviewer YAJd (W1) for the full setup and table. Finally, regarding the two author-readers in the human study, we recruited five additional external readers during the rebuttal period. With both authors completely excluded, the eight external readers prefer our trained library over the empty library by 77 wins to 28 losses (net preference +40.8%, 95% CI: [+20.8%, +59.2%], permutation test p = 0.003), and the professional screenwriter alone prefers it 10 to 2 (sign test p = 0.039). Please see our responses to Reviewers YAJd and rePM for the full breakdown.W2. The paper criticizes LLM-as-judge scoring but still relies on an LLM judge for five episode-level metrics.
We appreciate the opportunity to clarify this distinction. What we criticize in Section 4.1 is unanchored holistic scoring, where an LLM judge assigns an overall impression score from 1 to 5. As noted in Section 4.1, holistic scores are highly unstable and can swing from 2 to 4 on the exact same script across runs. In contrast, our five episode-level metrics reformulate judging into narrow, evidence-grounded event detection, four yes-or-no checks and one count of plot conflicts: the judge only checks whether a specific structural event occurred, such as whether the final scene cuts mid-action before the outcome lands, and is required to quote the exact script lines supporting its verdict. We verified empirically that this formulation eliminates the run-to-run instability of holistic scoring: across five independent re-evaluations of the same 385 scripts, the hard-cut ending rate varies by at most 1.3 percentage points and cold opens by at most 0.8 percentage points (test-retest Cohen's κ=0.94–0.97), and cross-checking our primary judge (Gemini 3.1 Pro Preview) against Claude Opus 4.8 yields 94% verdict agreement.
W3 & Q1. Key designs lack ablations. Would Opus with a strict but non-error-driven library, such as the +Distill library, match the gains?
This is a great falsification test, and we ran this exact experiment during the rebuttal period. We evaluated Claude Opus 4.8 across all 385 held-out episodes under three conditions: with an empty library (
Opus Base), with the 20-novel+Distilllibrary (131 strict rule cards distilled directly from the human scripts under the same 1,200-token linter limit), and with our 20-novel loop-trained library (23 rule cards).* denotes that the 95% paired bootstrap confidence interval against Opus Base excludes zero.
When Opus is given the non-error-driven
+Distilllibrary, 0 of the 6 directional metrics improve significantly and 2 actually degrade significantly (Unfilmable action linesandPlot conflicts). In contrast, with our loop-trained library, all 6 directional metrics improve significantly across the held-out novels. This directly confirms that the gains come from the error-driven reconstruction loop rather than rule strictness alone. The other components, including the 1,200-token card limit, the routing of cards by their descriptions, and the serialized micro-steps, are shared by +Distill and +Ours. Since the two methods differ only in the source of the learning signal, the advantage of +Ours over +Distill does not come from these shared components.