- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes an inference-time control method for creative writing through temporary context injection, inspired by the human writers' mind-wandering process. Specifically, a dictionary mapping certain words to passages is built before generation to provide content for temporary injection. During the generation process, an eligible dictionary word can trigger injection of its associated passage. A rule-based or trained router determines how much passage text to inject and how many new tokens to generate under the changed context. After this "wandering" process ends, the previous context will be restored, keeping the new story tokens and discarding the injected passage. Future tokens then continue to be generated based on the original context and the generated prefix so far. In experiments, the authors report that the proposed method improves Qwen2.5-1.5B's creativity and coherence over the temperature sampling and prompting baselines in one tailored story world, but shows little to no benefit at the 7B scale.
Critical strengths
-
The paper draws on creative incubation to suggest an interesting way to alter decoding without retraining the generator, which might be insightful for the relevant community.
-
The authors honestly report negative results and disagreements between evaluation methods.
Critical weaknesses
-
Limited Generalization and Impacts: A Highly-Tailored Setting and Small Models: The eight opening prompts and dictionary passages share one story world, and the study covers one model family with little demonstrated benefit at 7B. It is not clear how the demonstrated performance improvement would hold in more open-ended creative writing across different models. Also, there is no sufficient discussion on transferability and building cost of the word-passage dictionary, leaving concerns about how such a dictionary can be built for general domains.
-
Missing Human Calibration on Metrics: In the domain of LLM creative writing evaluation, there are many existing approaches from previous works [1, 2]. The proposed LLM-judge rubric needs validation against human judgments, leaving concerns about the reliability of the evaluation. The learned controller adds creativity and coherence in its reward (Eq. 3), and that weighting needs clearer justification when the objectives conflict (e.g., Figure 2 already shows metrics tradeoff analysis for the tested decoding methods).
-
Significant Computational Overheads: As the proposed method change the context, each wandering episode requires a fresh prefill (Algorithm 1, Section 3), which is not predictable and impact the major parallelism mechanism for normal LLM decoding. The resulting latency and computation overhead would increase significantly and needs to be measured. Without a matched-cost comparison, it is unclear whether the gains over simpler decoding justify the extra work.
-
The presentation overall is hard to follow: Several terms like "episodes", "second context", "step" are used without prior definition. The notation and the connection between the equations and the decoding procedure are also hard to follow. For example, the connection to [3] does not make the basic context injection operation easier to understand. A clearer walkthrough should connect the repeated episodes in Eq. 1, the number of passage sentences in Eq. 2, and the start and cooling rules. The "When to wander" part is particularly confusing. For the untriggered variant, a worked example would help readers follow how the reported waiting-time estimate sets the start probability and resulting amount of wandering.
References:
[1] Gómez-Rodríguez, Carlos, and Paul Williams. "A confederacy of models: A comprehensive evaluation of LLMs on creative writing." Findings of the Association for Computational Linguistics: EMNLP 2023.
[2] Wang, Yichen, et al. "Optimizing diversity and quality through base-aligned model collaboration." ICML 2026.
[3] Hinton, Geoffrey E. "Training products of experts by minimizing contrastive divergence." Neural computation 14.8 (2002): 1771-1800.
Questions for authors
-
How would you generalize the method and dictionary to open-ended creative writing beyond this story world?
-
How are the proposed metrics aligned with human judgments?
-
What is the extra inference cost compared to vanilla decoding, and do the quality gains hold at matched compute?
-
Please make the method clearer step by step, and explain what each experiment establishes.