acceptodds
Withdrawn from ICLR 2027

Spectral Invariants of Reasoning Trajectories: A Koopman‑DMD Dynamical View on LLM Internal Logic Geometry

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models produce multi‑step natural‑language reasoning chains, yet little is known about whether distinct logical inference types induce separable intrinsic geometric signatures within latent hidden‑state dynamics. Moving beyond static‑layer probing, we treat cumulative reasoning hidden‑state sequences as discrete phase‑space trajectories and adopt Koopman operator theory together with Exact Dynamic Mode Decomposition (DMD) to extract complex‑valued spectral fingerprints of internal reasoning dynamics. A major practical obstacle is that DMD eigenvalue sets from different reasoning traces vary in cardinality; naive zero‑padding introduces artificial mass at the origin and corrupts spectral comparisons. To address this, we develop a variable‑rank optimal‑transport metric for unaligned complex eigenvalue clouds, avoiding zero‑padding distortions. We state and empirically investigate the Spectral Invariant Hypothesis (SIH): spectral fingerprints from identical logic types exhibit smaller intra‑type distances than inter‑type distances. Evaluated on controlled logical reasoning benchmarks across multiple open‑weight models, our results confirm the hypothesis and reveal non‑monotonic scaling of dynamical manifold structure with model size. Our dynamical‑systems perspective provides a probe‑free diagnostic paradigm for analyzing LLM reasoning mechanisms.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Dynamical Systems, Representation Geometry, Logical Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_HBgFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper represents a reasoning trace as a sequence of step-level hidden-state summaries and uses dynamic mode decomposition to extract a spectral fingerprint. It compares fingerprints using a pairwise rank-truncated matching distance and studies whether spectra separate logical templates while remaining relatively stable across topics and languages. On Reasoning-Flow and five language models, the reported inter-logic to intra-logic distance ratios are approximately 2.37–2.67. Layer, rank, and context ablations are provided, together with results on orthogonal invariance and the effects of truncation.

Critical strengths
  1. Viewing reasoning traces through their spectral structure is an interesting direction and offers a compact way to summarize hidden-state trajectories.
  2. The invariance to orthogonal changes of coordinates is a useful property of the representation and is clearly motivated.
  3. The experiments cover several language models and include uncertainty estimates and extraction ablations. This gives the empirical observations some breadth.
Critical weaknesses
  1. I am not convinced that the proposed distance provides a sound basis for the claimed geometry. The pairwise truncation can discard meaningful spectral differences and even give zero distance to different spectra. Although the appendix acknowledges limitations, the main analysis still relies heavily on this construction. Its suitability needs stronger justification.
  2. The interpretation of the empirical results seems too strong. Separating logic templates does not by itself show that the method captures internal reasoning dynamics; template structure, trace length, and segmentation are plausible alternative explanations. The current controls do not sufficiently distinguish these possibilities.
  3. The added value of the dynamical analysis remains unclear. The traces are short, so the reported rank saturation is partly constrained by the number of available steps. Without convincing comparisons to simpler hidden-state summaries or controls that remove temporal order, I find the evidence for a specifically dynamical contribution limited.
Questions for authors
  1. Does the main separation persist when trajectories are matched for length and rank?
  2. How sensitive are the conclusions to replacing the truncated distance with full-spectrum optimal transport?
  3. How does the method compare with static hidden-state summaries, and what happens if the reasoning steps are shuffled?
Reviewer_44m4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether different types of logical reasoning in LLMs exhibit spectral invariants in hidden-state dynamics, using Koopman operator theory and Dynamic Mode Decomposition (DMD).

Critical strengths

The topic of investigating reasoning trajectories of LLM is interesting and timely. It helps us to better understand the mechanism of LLM, using a more physics manner.

Critical weaknesses

The procedure to get y1,..., y_T: MeanPool, HiddenState, seem not well explained.

A key assumption of the paper appears to be that the transition matrix A, across embeddings, is low-rank. It is not immediately clear why that holds. Since this assumption is central to the proposed methodology, its motivation and practical relevance should be more thoroughly discussed and justified.

The real-data analysis is not sufficiently convincing to support the paper’s broader claims about LLMs.

Questions for authors

I hope the authors can provide further clarification and justification for the critical weaknesses.

Reviewer_X29aOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether different types of logical inference leave distinguishable signatures in the hidden-state dynamics of language models during multi-step reasoning. For each reasoning trace, the hidden states of every step are extracted with the full preceding context, mean-pooled over the step's tokens, and stacked into a trajectory matrix. Exact Dynamic Mode Decomposition is applied to this matrix, with a per-trajectory rank chosen by a relative singular-value threshold, and the resulting set of complex eigenvalues is taken as a spectral fingerprint of the trace.

To compare fingerprints of different sizes, the paper truncates both eigenvalue sets to the smaller size after sorting by magnitude and solves an assignment problem between them, calling this a variable-rank Wasserstein-2 distance. It states a Spectral Invariant Hypothesis, namely that fingerprints of the same logic type are closer than fingerprints of different logic types, and gives a bound relating the variable-rank distance to a zero-padded variant, metric properties of the truncated distance, and invariance of the fingerprint to orthogonal transformations of the hidden space.

Experiments use the Reasoning-Flow benchmark, which instantiates thirty logical templates across topics and four languages, with three Qwen3 models from 0.6B to 8B parameters and Llama-3.1-8B. The ratio of inter-type to intra-type distances is between about 2.4 and 2.7 for all models and is significant under a label-permutation test. The paper further reports k-means clustering of per-type centroids, agreement of these clusterings across models, and sensitivity to the extraction layer, the number of steps, the rank cap and zero-padding.

Critical strengths

S1. Well-controlled data source and careful significance testing

The study builds on a benchmark in which the same logical template is rendered across many topics and several languages, which in principle separates logical form from surface content. The separation result is reported with bootstrap confidence intervals over logic types, a label-permutation test, and per-type checks, and it is examined across five models, four extraction depths, several prefix lengths and rank caps. The fingerprint is also shown to be invariant to orthogonal changes of the hidden-space basis, which makes cross-run comparison within a model meaningful.

Critical weaknesses

W1. Spectral fingerprint may not measure reasoning dynamics

The Koopman and DMD interpretation of the fingerprint rests on treating step-wise hidden states as snapshots of an autonomous dynamical system, but several properties of the construction make this interpretation uncertain. Each trace has at most sixteen steps while the hidden dimension is in the thousands, and the relative threshold of 10−1010^{-10} retains essentially every singular value, so the effective rank is in practice the number of steps minus one. The rank-cap sweep saturating at sixteen matches this bound rather than an intrinsic property of the dynamics. With so few snapshots in such a high dimension, the classical Krylov view of DMD[1] suggests that the eigenvalues are determined by how the last projected state is expressed through the earlier ones, rather than by a well-identified linear operator. Each step also appends new input text, so the trajectory is driven by external input at every step, a setting that standard DMD does not account for without an input term[2]. In addition, the motivation for the variable-rank distance states that Wasserstein distances require zero-padding when the two sets differ in size, whereas optimal transport between uniform empirical measures of different cardinalities is well defined without padding[3]. The proposed distance instead discards the smallest-magnitude eigenvalues, is not a metric for unequal ranks, and in the padding ablation yields lower separation than the zero-padded variant it replaces. Since the hypothesis is stated about the geometry of reasoning dynamics, it remains unclear what property of the model's computation the fingerprint and its distance capture.

W2. Separation is not attributed to logical structure beyond simpler features

The main evidence for the hypothesis is that traces of the same template are on average closer than traces of different templates, but the experiments do not rule out simpler explanations for this separation. In the benchmark, each logical template has its own number of steps and a shared step-wise phrasing such as repeated "Therefore" conclusions, so the number of eigenvalues and the structure of the step sequence already vary systematically with the logic type. No comparison is reported against fingerprints that use none of the dynamical machinery, for example the number of steps alone, distances between mean-pooled or per-step hidden states, or the same pipeline applied to shuffled steps or to an untrained model. The permutation test only shows that logic labels are associated with some property of the traces, which any of these features would also satisfy. Prior work on the same benchmark[4] has already reported that hidden-state trajectories are organised by logical structure rather than surface content, and the paper does not discuss what the spectral view adds to that finding. As a result, the evidence does not yet distinguish the Spectral Invariant Hypothesis from the possibility that the fingerprint mainly encodes trace length and template wording, which bears directly on the central claim of the paper.

[1]: Spectral Analysis of Nonlinear Flows

[2]: Dynamic Mode Decomposition with Control

[3]: Computational Optimal Transport

[4]: The Geometry of Reasoning: Flowing Logics in Representation Space

Questions for authors

Clarifications that could affect the assessment:

  1. Are the reasoning traces generated by each evaluated model, or are the benchmark's reference traces fed to every model? If the traces are shared, how do the authors interpret the cross-model agreement of clusters, given that the step structure is identical across models?
  2. What is the distribution of effective ranks per logic type, and how strongly does it correlate with the number of steps in each template? How large is the inter/intra ratio when traces are compared only within groups that have the same number of steps?
  3. How large is the inter/intra ratio obtained with simpler representations of the same traces, such as distances between mean-pooled hidden states or between step-aligned hidden states, and with the same pipeline applied to randomly permuted steps or to a randomly initialised model?
  4. How does the fingerprint relate to the dynamics-based interpretation when the trajectory is driven by new input at every step? Does a variant that accounts for the appended step input, or a standard Wasserstein distance between eigenvalue sets of different sizes, change the separation?

Suggestions for future work:

  • The paper presents the fingerprint as a probe-free diagnostic, but no diagnostic use is shown, such as relating fingerprints to reasoning errors or to model correctness. An experiment of this kind may help show the practical value of the representation.
  • The claim of non-monotonic scaling rests on the number of clusters selected by silhouette score over thirty centroids and on pairwise adjusted Rand indices, with silhouette values between 0.33 and 0.46. Reporting the stability of these clusterings may help readers judge how much weight the scaling statement can bear.
  • Minor presentation points: the hypothesis is written as "much smaller", while the observed ratio is about 2.5; the comparison of variable-rank and zero-padded strategies is repeated verbatim at the start of the theoretical analysis; a cross-reference reads "Equation equation"; and several related-work paragraphs, such as those on multimodal chain-of-thought and spatial reasoning models, are not connected to the question studied.
Reviewer_vUTaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hidden-state trajectories of large language models during multi-step logical reasoning from a dynamical-systems perspective. The authors construct context-cumulative reasoning trajectories, apply Exact Dynamic Mode Decomposition (DMD) to extract complex-valued spectral fingerprints, and introduce a variable-rank spectral distance for comparing trajectories with different numbers of eigenvalues. The paper proposes the Spectral Invariant Hypothesis (SIH): trajectories from the same logical type should have smaller spectral distances than trajectories from different logical types. The method is evaluated on the Reasoning-Flow benchmark using five open-weight models. The authors report inter-type to intra-type distance ratios between 2.37 and 2.67. The paper also includes permutation tests, spectral clustering, layer sensitivity, trajectory-prefix-length sensitivity, and DMD rank-cap ablations. Overall, the paper aims to develop a dynamical-spectral framework for studying geometric structure in the internal reasoning trajectories of language models.

Critical strengths
  1. A potentially useful dynamical perspective on reasoning representations. Applying DMD to hidden-state trajectories offers a complementary view to static representation analysis. The observed within-type versus between-type spectral separation provides an interesting empirical finding for further investigation.

  2. Multi-model evaluation with several robustness checks. The study covers five models and includes layer, trajectory-length, rank-cap, and distance-formulation ablations. These analyses provide useful evidence about the stability and scope of the reported separation.

Critical weaknesses
  1. The spectral separation needs stronger validation against distance-design effects. The variable-rank distance discards unmatched modes and can assign zero distance to distinct fingerprints, a trade-off acknowledged in Appendix D.2. Although the zero-padded baseline also shows separation, the paper does not establish how rank differences and discarded modes affect the inter/intra ratio. Comparing full eigenvalue distributions without truncation or zero-padding would strengthen the interpretation that the separation reflects logical structure.

  2. The main sampling and statistical protocols are insufficiently specified. The stated dataset design implies 2,400 sequences, rather than the reported 2,430, and the relationship between the 450-instance setting in Appendix C and the headline results is unclear. The sample-selection procedure, main extraction layer, and justification for unrestricted label permutation across related topical and language variants also need clarification. These gaps limit reproducibility and assessment of the statistical evidence.

Questions for authors
  1. What exact number of trajectories was used for each model, how were the samples selected, and what was the statistical unit? Please reconcile the 30×20×4=240030\times20\times4=2400 construction with the reported total of 2,430 sequences, explain the source of the extra 30 sequences, and explain how the 450-instance setting in Appendix C relates to the main results.

  2. How do effective-rank distributions and the magnitudes of discarded modes vary across logic types? Beyond the zero-padded baseline, does the separation persist when comparing the full eigenvalue clouds using standard optimal transport with normalized empirical weights, without truncation or zero-padding?

  3. Please specify the main experimental configuration, including model checkpoints, prompts and decoding settings, reasoning-step segmentation, numerical precision, and treatment of failed or incomplete traces. Were the headline results in Table 1 computed at layer L/2, as suggested by the identical ratios in Table 4? Please also justify whether unrestricted trace-level label permutation is appropriate given the shared templates and topical/language variants, or whether a stratified permutation test is needed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.