Prefix Semantic Dynamics in Long-Form Language Generation

Long-form responses from large language models (LLMs) can remain locally coherent while becoming semantically repetitive or increasingly detached from the original prompt as generation proceeds. We introduce the Prefix Semantic Dynamics (PSD) method to characterize and monitor this process during text generation. Specifically, PSD constructs a semantic trajectory from the normalized embeddings of successive cumulative text prefixes. This trajectory is further characterized by three geometric descriptors: correlation dimension $D_2$, semantic recurrence $\rho$, and tortuosity $\tau$, which describe its expansion, recurrence, and path geometry, respectively. We evaluate PSD on 720 text trajectories spanning natural, generated, and controlled text, where its geometric descriptors capture systematic differences in both semantic repetition and expansion. We then introduce PSD Control, which uses the evolving prefix trajectory as feedback to guide generation. Across 80 LongFact prompts, PSD Control consistently shifts text generation toward trajectories that are less recurrent, more expansive, and better aligned with the prompt than standard continuation. These results demonstrate that PSD provides a new representation that can both characterize semantic development and guide subsequent generation.