acceptodds
Under review as a conference paper at ICLR 2027

Prefix Semantic Dynamics in Long-Form Language Generation

Abstract

Long-form responses from large language models (LLMs) can remain locally coherent while becoming semantically repetitive or increasingly detached from the original prompt as generation proceeds. We introduce the Prefix Semantic Dynamics (PSD) method to characterize and monitor this process during text generation. Specifically, PSD constructs a semantic trajectory from the normalized embeddings of successive cumulative text prefixes. This trajectory is further characterized by three geometric descriptors: correlation dimension , semantic recurrence , and tortuosity , which describe its expansion, recurrence, and path geometry, respectively. We evaluate PSD on 720 text trajectories spanning natural, generated, and controlled text, where its geometric descriptors capture systematic differences in both semantic repetition and expansion. We then introduce PSD Control, which uses the evolving prefix trajectory as feedback to guide generation. Across 80 LongFact prompts, PSD Control consistently shifts text generation toward trajectories that are less recurrent, more expansive, and better aligned with the prompt than standard continuation. These results demonstrate that PSD provides a new representation that can both characterize semantic development and guide subsequent generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-form generation, text degeneration, controllable text generation, semantic trajectory, text embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K5iXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Prefix Semantic Dynamics (PSD), which represents an evolving response as a trajectory of cumulative-prefix embeddings and characterizes it using correlation dimension, semantic recurrence, and tortuosity. It further proposes a generation controller that uses these descriptors and prompt alignment to select among candidate continuations, with experiments demonstrating improvements in the targeted geometric metrics.

Critical strengths
  1. A useful process-oriented perspective. The paper examines how semantic content develops during generation rather than evaluating only the completed response. Cumulative-prefix trajectories provide a unified representation for both analysis and generation control.

  2. Relatively comprehensive analysis. The experiments include controlled text perturbations, sensitivity analyses, multiple generator–representation configurations, and cross-representation evaluation. These analyses help characterize the behavior of the proposed descriptors and clarify their dependence on measurement settings.

Critical weaknesses
  1. Insufficient evaluation of real-world writing quality. The paper does not evaluate performance on dedicated writing benchmarks, such as WritingBench or LongBench-Write. Although evaluation with a different representation model reduces concerns about using the same model for selection and evaluation, it still relies on a similar embedding-based metric framework. Consequently, the reported improvements do not adequately establish gains in coherence, informativeness, completeness, or overall writing quality.

  2. Incomplete assessment of computational cost. Generating four candidates per controlled round and encoding their extended prefixes introduces additional inference overhead. The reported latency ratio of approximately 1.33 in one configuration reflects parallel execution and should not be interpreted as only a 33% increase in total computation. Moreover, Table 12 shows that, with the 9B generator and Harrier representation, median runtime increases from 59.7 to 121.1 seconds, approximately doubling. The paper should report total generated tokens, representation-processing costs, or GPU time, and compare quality gains under matched computational budgets.

  3. Insufficient evidence for the necessity of the geometric descriptors. The current baselines do not establish whether the full PSD formulation offers substantial benefits over simpler candidate-selection criteria, such as prompt similarity or semantic novelty relative to previous content. Furthermore, TextOnlyRouter receives only the latest 900 words, making it an imperfect comparison with a method that uses cumulative-prefix history. Stronger information-matched baselines and descriptor-level ablations are needed to isolate the contribution of the proposed geometry.

Questions for authors
  1. Evaluate the method on dedicated writing benchmarks and include blinded assessments of coherence, semantic repetition, informativeness, and overall quality.

  2. Add simple relevance-and-novelty selection baselines and ablate each geometric descriptor, reporting both output quality and computational cost under matched budgets.

2 comments

We thank Reviewer K5iX for the specific comments. Our response to each point is below, with the experiments described in the general response.

Evaluation of writing quality. Of the qualities listed, semantic repetition is the one the controller is designed to reduce, and the revision evaluates it with a model outside the pipeline. PSD Control's responses reuse fewer already-active features than Continue's on the same prompts, in all 13 held-out domain groups (general response, part 1, Figure 4a). Content-word coverage of the prompt increases by 2.6–4.9 percentage points, and the factual-support audit excludes a loss larger than 0.057. Table 8 also applies PSD to LongBench-Write answers, to check that the measurement carries over to that benchmark.

Computational cost. We agree that the 1.33× ratio should not be read as total compute. It was wall-clock latency under parallel candidate generation. Table 20 reports about 34 generation calls per response versus 10 for Continue, 3.4–3.5× the generated tokens and 3.85–4.05× the representation tokens. PSD Control, PhaseAgnosticPareto and TextOnlyRouter all generate four candidates per controlled round, so they are compared at matched candidate budget.

Necessity of the geometric descriptors.

  • Information-matched router. The router's 900-word window holds the whole prefix through round 3 for every prompt and through round 4 for 71–90% of prompts. On those rounds PSD Control still has lower recurrence (by 0.029–0.062) and higher D₂ (by 0.036–0.109) than the router in all eight rollout–evaluator combinations, with every 95% CI excluding zero (Table 16, last block).
  • Without the phase rules. PhaseAgnosticPareto ranks the candidates on the descriptors and prompt alignment alone. It lowers ρ and raises alignment and D₂ relative to Continue in all eight combinations, with all 24 CIs excluding zero (Table 16).
  • Novelty criterion. On the perturbed texts, similarity to the nearest earlier segment has AUC 1.00 on repetition but 0.61 on reordering and 0.68 on drift, where ρ has 0.96 and 0.90 (Table 2). Feature reuse is itself a novelty criterion computed in an independent model, and within a round it is associated with Δρ (0.25–0.35, Table 9).

Thank you for your response. However, my main concerns about soundness remain unresolved, so I will maintain my original score.

Reviewer_wf6qOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Prefix Semantic Dynamics (PSD), which adapts the correlation-dimension framework of Du & Tanaka-Ishii (2025) from token-level next-token log-probability vectors to normalized embeddings of cumulative 100-token text prefixes x_{1:t_k}, summarizing the trajectory Z=(z_1,...,z_K) via finite-scale correlation dimension D2, semantic recurrence rho = C(epsilon*), and cosine tortuosity tau. Using a 720-trajectory diagnostic suite modeled closely on Du & Tanaka-Ishii (2025) (SEP, Gutenberg, Wikipedia, academic prose, Python code, shuffled/repetitive/bland/random-name/tree controls, and Normal/Bland/Incoherent/Repetitive prompt regimes), the authors characterize how cumulative-prefix geometry evolves across horizons and embedding models. They then introduce PSD Control and a phase-agnostic variant (PhaseAgnosticPareto) as training-free, closed-loop runtime controllers that generate four parallel 300-token candidate continuations per round (Continue, Branch, Densify, ReAnchor) and select one based on (D2, rho, tau) and prompt cosine alignment A_q. Across 80 shared LongFact prompts, two Qwen3.5 generators (9B and 122B-A10B-FP8, run with thinking disabled), and two representations (Qwen3-Embedding-8B and Harrier-OSS-v1-27B), both selectors consistently shift internal trajectory geometry (rho, D2, A_q), while yielding modest gains in prompt cosine relevance (+0.01-0.03) and prompt-stem coverage (+2.6%-4.9%), mixed changes in exact sentence repetition, and no significant improvement in LongFact factual support.

Critical strengths

1: PSD offers a training-free, black-box, closed-loop monitoring and control without logit access or RL post-training, making the method very uniform and easy to reproduce or apply to any models.

Critical weaknesses

1: The paper seems not targeting any meaningful downstream tasks and just optimising the parameters ρ, A_q and D_2 of the generation results it introduces. It does not test on any human-visible targets and does not display any difference in its generation examples, making it hard to verify what PSD actually does.

2: PSD targets non-thinking models only, reasoning models have to disable the "thinking" mode to use PSD, while many long-form generation tasks can be achieved via thinking/reasoning.

Questions for authors

1: Could you offer some examples on PSD generated texts versus non-PSD and texts generated via reasoning, as long as some human-understandable metrics like accuracy, consistency, or human-rated generation quality etc.?

2: PSD seems somehow similar to the "Pro" reasoning mode of GPTs, which can really improve generation quality via parallel reasoning/generation. How does the generation result of "pro reasoning" compare with PSD?

2 comments

We thank Reviewer wf6q for the careful reading. We address each question below, and the general response describes the experiments in full.

What PSD does and its downstream task. PSD measures how the content of a long text develops using cumulative-prefix embeddings, without access to the generating model. Its task is monitoring: detecting when a long text starts to repeat itself verbatim, repeat itself in different words, drift off topic, or lose its order. The new controlled-perturbation test evaluates this task directly on known edits. On human passages ρ detects the four changes with AUCs of 0.86, 0.84, 0.90 and 0.96, while each lexical or segment-level statistic falls to 0.50–0.61 on at least one of them (Table 2). For the controller, a model outside the pipeline measures less feature reuse in PSD Control's responses than in Continue's (Figure 4a).

Examples and human-understandable metrics. Table 18 shows Continue and PSD Control side by side on a randomly drawn prompt, with ρ, the selected action and the opening words of every round. From round 3 on, Continue's ρ stays at 0.49–0.61 and PSD Control's at 0.36–0.42, and the re-anchoring rounds (3, 6, 9 and 10) return to the question in the prompt. Content-word coverage of the prompt increases by 2.6–4.9 percentage points in all four configurations.

Thinking models. PSD does not require thinking to be disabled, since it is computed on the visible answer. We ran Qwen3.5-9B with thinking on and off on the LongBench-Write prompts that request at least 1,000 words (Table 8). On the 14 of 30 prompts where both modes reached 30 prefixes, recurrence is comparable (ρ = 0.53 with thinking and 0.49 without, p = .25). The recurrence PSD measures is therefore also present in the answers of a reasoning model.

"Pro" reasoning mode. The closest design we could test is TextOnlyRouter, which generates the same four candidates per controlled round and lets the generator choose among them. PSD Control has lower recurrence and higher prompt alignment and D₂ than this router in all eight rollout–evaluator combinations, four of which use the representation not used for selection (Table 16). This also holds in the rounds where the router has the whole prefix in context. In addition, PSD can be computed on the output of any such system.

Reviewer_wf6qOpenReview ↗

Thanks for the rebuttal.

Thanks for the rebuttal. I'll keep my original scoring.

Reviewer_r2JBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Prefix Semantic Dynamics (PSD), which constructs semantic trajectories from embeddings of cumulative text prefixes and characterizes the evolution of long-form text generation using three geometric descriptors: correlation dimension , semantic recurrence , and trajectory tortuosity. Building on PSD, the authors further propose PSD Control, which employs four prompt-driven generation strategies to produce candidate continuations and selects the next text segment based on the semantic states of the candidate-extended trajectories.

Critical strengths

S1. Interesting Research Problem. The paper addresses semantic repetition and drift in long-form language generation and investigates how semantic evolution can be monitored and controlled during generation. S2. Novel Semantic Trajectory Representation. The paper introduces Prefix Semantic Dynamics (PSD), which characterizes semantic evolution through cumulative-prefix embeddings and three interpretable geometric descriptors: correlation dimension, semantic recurrence, and trajectory tortuosity. S3. Effective Generation Control. The proposed PSD Control framework uses semantic trajectory feedback to guide candidate selection, consistently reducing semantic recurrence while improving semantic expansion and prompt alignment across the evaluated configurations.

Critical weaknesses

W1. Limited Independent Evaluation of Generation Quality. The main improvements are measured using geometric descriptors that are also employed by the proposed controller. Although PSD Control improves semantic recurrence, expansion, and prompt alignment, it remains unclear whether these improvements consistently translate into better overall generation quality, particularly factual accuracy and coherence. W2. Insufficient Validation of the Proposed Semantic Representation. The paper does not sufficiently demonstrate why cumulative-prefix trajectories are preferable to simpler alternatives, such as segment-level embeddings or direct semantic similarity measures. Additional comparisons would help establish the necessity of the proposed geometric descriptors. W3. Potential Bias from Overlapping Prefixes. Since consecutive cumulative prefixes share substantial textual content, their embeddings may naturally exhibit high similarity. It remains unclear to what extent the measured semantic recurrence reflects actual content repetition rather than structural overlap between prefixes. W3. Additional Inference Overhead. PSD Control requires generating and evaluating multiple candidate continuations at each controlled generation round, introducing additional computational costs. The paper would benefit from a more systematic analysis of the trade-off between generation quality and inference efficiency. W4. Additional Inference Overhead. PSD Control requires generating and evaluating multiple candidate continuations at each controlled generation round, introducing additional computational costs. The paper would benefit from a more systematic analysis of the trade-off between generation quality and inference efficiency.

Questions for authors
  1. Could the authors provide an independent evaluation of generation quality, particularly factual accuracy and coherence, beyond the geometric metrics used by the proposed controller?

  2. Could the authors compare the proposed semantic representation with simpler alternatives and analyze whether overlapping prefixes bias the semantic recurrence measurements?

  3. Could the authors quantify the inference overhead of PSD Control and evaluate the trade-off between generation quality and computational efficiency?

1 comment

We thank Reviewer r2JB for the constructive review. Brief answers follow, with the experiments described in the general response.

Independent evaluation of generation quality. We agree that the controller should not be evaluated only with its own descriptors. The revision adds an evaluation with a model outside the pipeline, which finds less feature reuse in PSD Control's responses than in Continue's in all 13 held-out domain groups (general response, part 1, Figure 4a). On factual accuracy, the method-blinded audit excludes a loss of factual support larger than 0.057. Content-word coverage of the prompt also increases in all four configurations.

Simpler alternatives. Table 2 compares ρ and D₂ with segment-level embedding measures and lexical statistics on the same perturbed texts (general response, part 2). Each simpler measure is weak on at least one edit. Nearest-earlier similarity has AUC 0.61 on reordering, and distinct-2 has 0.52 on paraphrased repetition. ρ is the only single measure whose worst-case AUC exceeds 0.8 on the human passages (0.84).

Overlapping prefixes. This is a fair concern, and we tested it directly. Every perturbed variant has the same number of prefixes and the same prefix overlap as its original, yet ρ rises by 2.0 SD under reordering and falls by 1.6 SD under topic drift. Overlap alone therefore cannot explain these differences.

Inference overhead. Table 20 now reports calls and tokens: about 34 generation calls per response versus 10 for Continue, with 3.4–3.5× the generated tokens. The 1.33× ratio in the submission was wall-clock latency under parallel candidate generation. Monitoring alone requires only embedding the prefixes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All

We thank the reviewers for their detailed feedback. The reviews share four questions: whether the controller's effect holds under an evaluation independent of our descriptors, whether ρ reflects genuine content changes rather than prefix overlap, whether simpler statistics would suffice, and what the controller costs. We ran new experiments on each and uploaded a revised PDF. The revision presents PSD primarily as a text-level diagnostic that needs no access to the generating model, and PSD Control as a proof of concept that this state can guide generation.

1. Independent evaluation (Figure 4, Tables 9–10). We evaluated the outputs with Qwen3.5-27B, which plays no role in generation, selection or embedding, through its released sparse autoencoder (Qwen-Scope). For each segment we measure feature reuse, the fraction of its activation on features already active earlier in the response. All comparisons were fixed in advance in a hash-locked registry on 26 development prompts and run once on 25 held-out prompts (13 domain groups).

  • Controller. On the same prompts, PSD Control's responses reuse fewer features than Continue's (segment novelty +0.024, 95% CI [0.018, 0.030], p < .001, same sign in all 13 groups).
  • Descriptors. Within a round, segments with higher feature reuse raise ρ more (standardized association 0.25, and 0.35 among the four candidates), after controlling for length, word repetition and step.
  • Content vs. wording. In a matched-wording experiment on 192 new texts, continuations that add content were matched to restatements in word repetition (within 0.021). Adding content lowered ρ by 0.19–0.24, while changing word repetition by 0.20 under a fixed instruction moved ρ by 0.02.
  • Other output measures (Section 5.4). Content-word coverage of the prompt, an embedding-free count, increases by 2.6–4.9 percentage points in all four configurations. The method-blinded factual-support audit estimates a difference in support rate of +0.024 (95% CI [−0.057, 0.105]), which excludes a loss larger than 0.057.

2. Controlled-perturbation test (Section 5.3, Table 2, Appendix Tables 5–7). We altered the later part of 80 texts (40 human passages, 40 Qwen3.5-122B responses) in four known ways, each at three doses: verbatim repetition, paraphrased repetition, topic drift and reordering. Every variant keeps the same number of prefixes and the same prefix overlap as its original, so overlap cannot produce differences between variants. The table shows the human passages at the highest dose: shift in SD, with detection AUC in parentheses.

ρD₂seg. mean sim.nearest earlier seg.adjacent seg.distinct-2rep. 4-grams
verbatim repetition+1.37 (.86)−0.93 (.77)+0.89 (.74)+4.34 (1.00)−0.19 (.57)−8.38 (1.00)+27.8 (1.00)
paraphrased repetition+1.27 (.84)−0.63 (.68)+0.80 (.72)+3.19 (1.00)−0.22 (.58)+0.10 (.52)+0.55 (.70)
topic drift−1.57 (.90)−0.78 (.75)−2.30 (.95)−0.57 (.68)−0.34 (.62)+0.72 (.72)−0.37 (.63)
reordering+2.01 (.96)−0.79 (.75)0.00 (.50)−0.41 (.61)−3.15 (.97)0.00 (.50)−0.01 (.51)
worst-case AUC.84.68.50.61.57.50.51

With overlap fixed, ρ rises by 2.0 SD under reordering and falls by 1.6 SD under topic drift. Paraphrased repetition moves ρ almost as much as verbatim repetition (+1.27 vs. +1.37 SD), while repeated 4-grams retain only 2% of their verbatim shift. On the human passages ρ is the only single measure whose worst-case AUC exceeds 0.8, and each simpler statistic falls to 0.50–0.61 on at least one edit. On the Qwen3.5-122B responses ρ shifts in the same directions under verbatim repetition and topic drift, and reordering of Qwen text is detected by τ or D₂ (Tables 5 and 6).

3. Cost (Table 20). The 1.33× ratio in the submission was wall-clock latency under parallel candidate generation. In compute, the three selection policies each make about 34 generation calls per response versus 10 for Continue, generate 3.4–3.5× the tokens and send 3.85–4.05× the tokens to the representation model. Monitoring alone requires only embedding the prefixes.

Other changes. Table 8 reports a thinking-mode run, Table 16 an information-matched comparison with TextOnlyRouter, and Table 18 a side-by-side example. Figure 4 is new, and the former Figure 6 is now Figure A13.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.