acceptodds
Under review as a conference paper at ICLR 2027

LEXIS: A Symbolic Vocabulary for Time Series Understanding in Pretrained Language Models

Abstract

Pretrained large language models (LLMs) exhibit strong language understanding and reasoning capabilities, but applying these capabilities to time series understanding requires an interface that exposes temporal structure in a form legible to the model. Existing interfaces represent a series either as decimal strings or as embeddings produced by a series encoder. Decimal strings preserve the precise values but obscure their temporal relationships, while encoder embeddings are compact and information rich but lack semantics known to an LLM. We propose , an interface that expresses a time series as language by extending the vocabulary of a pretrained LLM. A small set of new symbols describes the structure of a series, each symbol denoting one known component of a curve, and their embeddings are derived from the geometric relationships among the symbols. Across three time series understanding benchmarks, LEXIS improves a Qwen3-4B backbone by 31.09%, 13.40%, and 4.45% and surpasses evaluated general-purpose LLMs and time series language models on most tasks. Controlled comparisons show that LEXIS outperforms the decimal and encoder embedding representations under the same training recipe. Ablations show that the model effectively derives temporal properties from the LEXIS representation and benefits consistently from the symbol geometry. These results establish LEXIS as an effective and principled interface between time series and pretrained LLMs.

Primary area:
learning on time series and dynamical systems
Keywords:
time series understanding, time series LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ueDQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LEXIS, an interface that represents time series through patch statistics and compositional shape tokens. It adds 50 shape symbols to Qwen3-4B, initializes their embeddings using axis/level geometry, and trains the model through series–text alignment. The model outperforms models trained with decimal inputs on three time-series understanding benchmarks. Component interventions are used to examine which parts of the representation the model relies on.

Critical strengths
  1. The representation is compact and decodable: each shape token has a known contribution to the curve. It uses roughly one quarter of the tokens required by decimal serialization.
  2. The experiments go beyond aggregate accuracy. Masking and shuffling show that the model uses shape information and temporal order. Against the trained decimal baseline, LEXIS gains 3.36, 2.41, and 2.91 percentage points on the three benchmarks.
Critical weaknesses
  1. The initialization ablation changes both geometry and scale. New embeddings are placed at the typical vocabulary radius, while the baseline starts near the mean. As Appendix A.4 explains, this also changes their initial accessibility to the output head. Table 3 therefore leaves open whether the gain comes from meaningful symbol geometry or simply better-scaled initialization.
  2. The training comparison needs clearer accounting. Appendix B describes the same examples, one epoch, and a common packed-token schedule, despite large differences in sequence length. Actual steps and example exposure for each variant would resolve this ambiguity. The soft-token baseline also omits series-writing examples, so its comparison includes a difference in supervision.
Questions for authors
  1. Does geometric initialization still help against a radius-matched control with shuffled symbol-to-position assignments?
  2. Please report actual steps, tokens, and example exposure for each representation, including how long examples are handled.
  3. How are the coordinate error and free-share metrics in Table 6 defined? The reported 0.6% coordinate/radius deviations and 14.7% orthogonal squared-norm share appear difficult to reconcile under the centered definitions in Appendix A.4.
2 comments

We thank the reviewer for their time and thoughtful feedback and address the concerns below.

R1: Geometry or scale of the initialization

Response to W1 and Q1

We run the radius-matched control, which places the symbols on the same shell (same norm as the pretrained vocabulary) with random coordinates, following the same training recipe.

VariantRadiusGeometryLeashTSAQATSETB
Mean initializationnot matchednono72.7256.5736.51
Random coordinates (new)matchednono74.0957.6937.65
Geometric initialization, no leashmatchedyesno76.9158.9837.51
LEXIS-4Bmatchedyesyes77.0960.7238.06

The control separates the two effects (TSAQA / TSE / TB):

  1. Scale. Matching the radius alone adds +1.37 / +1.12 / +1.14 over mean initialization.
  2. Geometry. At the same radius, the geometric coordinates add a further +2.82 / +1.29 / −0.14 over random coordinates, and +3.00 / +3.03 / +0.41 together with the leash.

Both therefore contribute: scale gives about 1 point on every benchmark, and geometry gives most of the gain on TSAQA and TSE, while the gain on TB comes from scale. This is consistent with Section 4.4 of the paper, and we will add the control to the revision.

R2: Training accounting

Response to W2 and Q2

The corpus has 857,912 stage 1 (s1) and 478,233 stage 2 (s2) samples. Samples are packed into sequences of 4,800 tokens, each stage is one epoch, and samples longer than the sequence length are dropped at export (evaluation has no cutoff).

VariantSamples s1 / s2Steps s1 / s2Input tokens s1 / s2
LEXIS856,799 / 477,841 (1,113 / 392 dropped)9,325 / 4,392622M / 293M
Decimal731,713 / 440,957 (126,199 / 37,276 dropped)19,745 / 11,115≈1.33B / 0.75B
Soft token420,000 / 395,824 (series-writing samples excluded)1,197 / 896≈80M / 60M
Envelope-only857,892 / 478,233 (20 / 0 dropped)5,366 / 2,547≈412M / 196M

Soft-token supervision. The soft-token baseline cannot emit a series and is trained only on the samples whose target is text. As a direct control we trained LEXIS-4B on exactly those samples (reading rows only): 76.18 / 59.92 / 35.79 against 71.73 / 55.90 / 37.65 for soft tokens, so the lead on TSAQA and TSE holds at equal supervision (+4.45, p=0.0003p = 0.0003; +4.02, p=0.039p = 0.039).

R3: Definitions in Table 6

Response to Q3

Each symbol embedding EE is centered on the pretrained mean mm. UU holds the 26 placement directions, κ\kappa is the scale, ϕ\phi the target coordinates and RR the pretrained radius.

QuantityDefinition
radius∥E−m∥/R\lVert E - m \rVert / R
coordinate error∥κ−1U⊤(E−m)−ϕ∥/∥ϕ∥\lVert \kappa^{-1} U^{\top}(E - m) - \phi \rVert / \lVert \phi \rVert
free share∥(E−m)−UU⊤(E−m)∥/∥E−m∥\lVert (E - m) - U U^{\top}(E - m) \rVert / \lVert E - m \rVert, median over the 50 symbols

Free share in Table 6 is a ratio of norms, which the caption and the text mislabel as a share of squared norm. As a share of squared norm it is 2.2% for LEXIS (0.1472≈0.0220.147^2 \approx 0.022), which is consistent with the 0.6% coordinate and radius deviations. We will correct the caption and the text.

We thank the reviewer again for their feedback that strengthens this work.

Dear Reviewer ueDQ,

Thank you again for your careful review. We appreciate that you found our representation compact and decodable, and found that our experiments go beyond aggregate accuracy to show how the model uses shape information and temporal order.

In our response, we added a radius-matched control that separates the geometry of the initialization from its scale (R1), reported the samples, steps and input tokens of every trained variant together with a supervision-matched control for the soft-token baseline (R2), and gave the definitions behind Table 6 with a correction of its free share label (R3).

Your comments have made this work stronger, and we would be delighted to discuss any remaining questions. Please do not hesitate to contact us for further discussion.

Authors

Reviewer_6G8pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LEXIS represents time series using local statistics, discrete shape symbols, and a separate transient component. The symbols have a fixed decoder and are added to a pretrained language model through two-stage training. On three benchmarks, the Qwen3-4B version outperforms trained decimal, soft-token, and statistics-only baselines. Ablations test the use of shape information and the proposed initialization.

Critical strengths

The motivation is clear: decimal strings leave temporal structure implicit, while projected representations lack meanings familiar to the language model. LEXIS represents series through a compact set of compositional symbols, each with a known referent, and uses their geometry for initialization and training. The matched comparisons and component-wise interventions help examine how the model uses these symbols. The method and experiments are clearly described and easy to follow.

Critical weaknesses

The vocabulary makes temporal structure explicit, but I am less convinced that the experiments show how it addresses the claimed difficulty of connecting time series to a pretrained language model. The new symbols still need to be learned, and having a fixed decoder does not by itself show that their meanings are easier for the language model to learn. The gains could come from the temporal decomposition and paired training rather than the symbolic vocabulary. The current baselines do not separate these explanations: they differ in the information presented to the model, and the soft-token baseline also lacks the series-generation supervision used by LEXIS. A comparison that holds temporal information and supervision fixed would better support the motivation. Several gains are also small enough that uncertainty across runs matters.

Questions for authors

What evidence shows that the symbolic vocabulary helps the language model learn the connection between time series and text, beyond the benefit of explicitly providing temporal structure? How does LEXIS compare with other representations of the same decomposed information under the same supervision? Please also report uncertainty for the main comparisons so that the smaller gains can be assessed.

3 comments

We thank the reviewer for their time and thoughtful feedback. The review asks for a comparison holding the decomposition and the supervision fixed, and for uncertainty. We address these concerns in four parts.

R1: Symbolic vocabulary vs explicit temporal structure

Response to W, Q1 and Q2

We train Qwen3-4B with the same recipe and the same supervision (series writing included) as LEXIS-4B. All variants keep the decomposition and differ only in the shape representation. Lettered shape levels and random coordinates are new ablation runs; the matched RVQ-VAE is a new learned-tokenizer baseline.

VariantControlTSAQATSETB
LEXIS-4Breference77.0960.7238.06
Envelope-onlysame decomposition, shape not given72.1857.2433.02
Lettered shape levelssame decomposition, shape levels written with existing letter tokens (no new token)75.4557.7734.69
Random coordinatessame new symbols, embeddings placed at random on the shell (no geometry)74.0957.6937.65
Matched RVQ-VAEsame number of new symbols, meanings learned by a residual VQ-VAE (no fixed referent)75.3658.9838.64
  1. Same information, same supervision. The lettered variant gives the model exactly the information LEXIS gives, under the same supervision, with tokens it already knows. LEXIS-4B is ahead by 1.64 / 2.95 / 3.37 (TSAQA / TSE / TB), so the gain is not explained by the decomposition and paired training alone.
  2. Symbol geometry facilitates the learning of new symbols. New symbols indeed still need to be learned. With random embeddings the same symbols are no better than letters on TSAQA and TSE, and the geometric initialization helps the learning of new vocabulary, adding +3.00 / +3.03 / +0.41 over random placement.
  3. Analytic against learned symbols. The matched RVQ-VAE replaces the fixed symbols with the same number of learned ones. LEXIS-4B is ahead by 1.73 / 1.74 on TSAQA and TSE and 0.58 behind on TB, with no tokenizer training, and its symbols have meanings known in advance, which the analyses in R2 rely on.

In addition, the fixed decoder makes the representation auditable, which enables the interventions in Section 4.3 of the paper.

R2: Connection between symbols and language

Response to Q1

R1 shows the benefit in accuracy. Here we test the connection directly, with two new analyses of the trained LEXIS-4B. Both ask whether the model has linked a single symbol to the words for the property that symbol denotes.

1. Does a symbol evoke the right words inside the model? We use the logit lens (nostalgebraist, 2020), which reads an intermediate hidden state through the output head of the model, to diagnose which words a symbol evokes. On 1,008 series crops, half rising and half falling, we compare at the position of the slope symbol the log probability the model gives to words for falling and to words for rising.

  • On the falling crops, the model gives 5.32 nats more log probability to falling words relative to rising words than on the rising crops, peaking at layer 18.
  • The effect is outside a null of 2,000 random word groups at every layer from 16 to 28, and it is absent before training.

Training has therefore mapped the new symbol onto vocabulary the language model already had, which is the connection between series and text that the reviewer asks about, measured for one symbol.

2. Does the model answer from the symbols? We use a counterfactual input intervention to diagnose whether the answers depend on the symbols. On 105 questions about a turning point (peak or trough), each series spans two patches. We exchange between the two patches the symbols of the first two shape axes (axis 1, slope, and axis 2, curvature), and leave the other six axes and every numeral unchanged. By the fixed meaning of these two axes, the exchange turns a peak into a trough.

  • Correct answers fall from 81 to 8 of 105: the model now reports the turning point that the swapped symbols describe.

The answers therefore depend on the symbols, in the direction their meanings predict, and not only on the explicit statistics given beside them.

Both tests require symbols with fixed referents: we know in advance which words a symbol should evoke and how an answer should change when it is swapped. A learned codebook or a soft token gives no such prediction to test.

R3: Supervision of the soft-token baseline

Response to W (series-generation supervision)

For the soft-token comparison we ran the direct control: LEXIS-4B trained on exactly the rows the soft-token baseline uses, those whose target is text (reading rows only).

VariantTSAQATSETB
Soft token71.7355.9037.65
LEXIS-4B, reading rows only (new)76.1859.9235.79
LEXIS-4B77.0960.7238.06

At equal supervision LEXIS keeps its lead over soft tokens on TSAQA and TSE (+4.45, p=0.0003p = 0.0003; +4.02, p=0.039p = 0.039).

R4: Uncertainty

Response to W (small gains) and Q3

We report paired bootstrap 95% CIs and exact McNemar tests for the comparisons of Table 1:

LEXIS −TSAQATSETB
decimals+3.36 [0.82, 5.91], p=0.01p = 0.01+2.41 [0.69, 4.13], p=0.006p = 0.006+2.91 [0.95, 4.90], p=0.005p = 0.005
soft tokens+5.36 [3.00, 7.82], p<10−4p < 10^{-4}+4.82 [1.21, 8.58], p=0.01p = 0.01+0.41 [−1.45, 2.27], p=0.70p = 0.70
envelope-only+4.91 [2.55, 7.27], p<10−4p < 10^{-4}+3.48 [0.70, 6.26], p=0.014p = 0.014+5.04 [2.16, 7.92], p=0.001p = 0.001

We additionally train LEXIS-4B with two other seeds and report the mean and standard deviation across runs.

SeedTSAQATSETB
first (reported)77.0960.7238.06
second78.3659.9239.28
third77.0059.9241.46
Mean ± sd77.48 ± 0.7660.19 ± 0.4639.60 ± 1.72

We will add these results to the revision.

We thank the reviewer again for their feedback that strengthens this work.

Dear Reviewer 6G8p,

Thank you again for your thoughtful review. We appreciate that you found our work clearly motivated, built on a compact set of compositional symbols with known referents, and supported by matched comparisons and component-wise interventions that are easy to follow.

In our response, we compared LEXIS with other representations of the same decomposed information under the same supervision (R1), added two analyses of how the model connects the symbols to language (R2), matched the supervision of the soft-token baseline (R3), and reported confidence intervals and additional training seeds (R4).

Your comments have made this work stronger, and we would be delighted to discuss any remaining questions. Please do not hesitate to contact us for further discussion.

Authors

Reviewer_FYHBOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LEXIS, an encoding mechanism for numerical time series signals for use in LLMs. It separates local level and scale from shape, represents shape using quantized PCA coefficients, and introduces a small vocabulary of symbols with structured embeddings. The method is evaluated on three time series understanding benchmarks using a Qwen3 4B backbone.

Critical strengths
  • Representing time series effectively for LLMs is an important unsolved problem.
  • The representation is compact and interpretable, with a deterministic decoder and clearly defined components.
  • The paper includes useful trained baselines and ablations examining how the model uses the representation.
  • The paper is generally clear, and the appendix provides substantial implementation detail.
Critical weaknesses
  • Comparisons with learned discrete representations, such as from (residual) vector-quantized VAEs, are important and missing. These approaches would also provide fixed vocabularies with many of the same benefits, but can be learned more directly from the data and have been used successfully in other domains such as audio. They could offer better fidelity or different compression tradeoffs, but the paper does not establish how LEXIS compares with them.
  • The reconstruction error appears to be fairly large (table 5). An NRMSE of roughly 0.15 or an R2 of <0.9 is actually a pretty substantial amount of error for time series representations as many tasks require strong numerical fidelity. Preserving broad shape may be sufficient for some questions, but precise values, fine details, threshold crossings, and brief events can matter substantially. The transient channel helps preserve spikes, but this is a significant limitation. Also, the paper does not adequately establish when the remaining approximation errors affect downstream answers.
  • The representation quality vs size tradeoff is insufficiently evaluated. Figure 7 plots reconstruction performance as the number of axes and patch length change, but does not connect these choices to downstream understanding accuracy. For a paper centered on effective encoding, the relationship between representation size, reconstruction fidelity, and task performance is central.
  • The choice of 50 shape symbols and the quantization granularity per PCA axis is not sufficiently justified. The appendix briefly discusses alternative allocations at a fixed bit budget, but does not establish whether larger vocabularies improve performance. Small changes in reconstruction error or retained energy can still matter considerably for particular tasks.
  • The experiments do not sufficiently isolate the source of the improvements. Preprocessing, representation, embedding initialization, and training objectives differ across comparisons. It remains unclear how much of the benefit comes specifically from the symbolic vocabulary and its proposed geometry.
  • It is unclear how well this method generalizes. The paper does not establish transfer to unseen domains or source datasets, and all trained results use the same backbone LLM.
  • The contribution is primarily an empirical combination of established techniques. The gains over trained baselines are relatively modest, their reliability is not clearly reported, and the missing tokenizer comparisons leave insufficient evidence of a substantial advance.
Questions for authors
  • How does LEXIS compare with learned tokenizers such as (residual) VQ-VAEs?
  • How do representation size and reconstruction fidelity effect downstream understanding accuracy?
  • What happens with larger vocabularies or finer quantization of the PCA coefficients?
  • How well does this representation work for answers that depend on precise values, threshold crossings, or brief events?
  • Does the encoder transfer to domains and source datasets outside the training distribution?
5 comments

We thank the reviewer for their thoughtful feedback. In response, we ran new experiments: two RVQ-VAE tokenizers (R1), accuracy as shape axes are removed (R3), a 92 symbol alphabet (R4), controls for each confounded factor (R5), and additional training seeds (R7). Each concern is addressed below.

R1: Learned RVQ-VAE tokenizer

Response to W1 and Q1

We train two RVQ-VAE tokenizer baselines with the training recipe of the paper.

  1. Matched RVQ-VAE: 8 residual codebooks with 50 symbols in total, keeping the envelope and transient block, so that only the shape symbols are replaced by the same number of learned tokens.
  2. Conventional RVQ-VAE: a TOTEM-style convolutional encoder with two codebooks of 256 codes (512 codes in total), without the LEXIS structure (no envelope, no transient block).
TokenizerAvg. Recon. NRMSETSAQATSETB
Matched RVQ-VAE0.21075.3658.9838.64
Conventional RVQ-VAE0.12473.8259.3435.60
LEXIS0.17477.0960.7238.06

LEXIS matches or exceeds both learned tokenizers: it is ahead on TSAQA and TSE, and the matched RVQ-VAE is 0.58 ahead on TB. The conventional RVQ-VAE reconstructs best and is behind LEXIS on all three benchmarks, so lower reconstruction error does not translate into better understanding (see also R2 to R4).

LEXIS reaches this accuracy with no tokenizer training, and its symbols have fixed meanings along the axis ladder, which enables the per-axis analysis of Section 4.3. An RVQ-VAE offers neither.

We will add these results to the revision.

R2: Reconstruction error and accuracy

Response to W2 and Q4

We acknowledge that the representation is lossy. LEXIS targets the understanding of temporal features rather than point-value precision, and this loss of precision is a common limitation of quantization-based representations.

To establish when the error affects answers, we vary it in both directions around the reported model:

  1. Lower error: a LEXIS-4B variant trained with 92 symbols instead of 50, quantizing each shape axis more finely at the same 8 tokens per patch (see also R4).
  2. Higher error: the reported model evaluated with only the first KK of its 8 shape axes, the remaining axes neutralized at inference (see also R3).

The table reports the difference from the reported model (exact McNemar pp in parentheses).

RepresentationAvg. Recon. NRMSETSAQATSE
LEXIS, 92 symbols0.137+0.64 (0.59)−1.07 (0.55)
LEXIS, 50 symbols (reported)0.17477.0960.72
6 of 8 shape axes0.191−0.55 (0.44)−0.47 (0.34)
4 of 8 shape axes0.227−1.36 (0.09)−0.84 (0.16)
2 of 8 shape axes0.360−2.45 (0.01)−1.67 (0.008)
no shape axes0.696−10.09 (<10−4<10^{-4})−4.40 (<10−4<10^{-4})

Lowering the error by 21% does not improve TSAQA or TSE, and raising it to 0.227 causes no significant drop; accuracy falls significantly only from 0.360. A tokenizer with a lower error is also not more accurate (R1). The reported model therefore operates where reconstruction error does not limit understanding accuracy.

Questions that require an exact value at a single step are outside what the representation targets. A lossless input is the better choice there, and we will state this limitation in the revision.

R3: Representation size and accuracy

Response to W3 and Q2

The table in R2 connects representation size, reconstruction error and accuracy: it evaluates LEXIS-4B with only the first KK of its 8 shape axes, the remaining axes neutralized at inference.

Accuracy falls as axes are removed, with diminishing returns: the lower axes contribute most of the accuracy. This agrees with the rail analysis of Section 4.3. Because the model was trained with 8 axes, these numbers are a lower bound for a model trained at a smaller KK. We will add these results and a discussion to the revision.

R4: Larger representation and finer quantization

Response to W4 and Q3

We train a variant of LEXIS-4B with 92 symbols, refining the ladder to (15,15,13,11,11,9,9,9)(15, 15, 13, 11, 11, 9, 9, 9) at the same 8 tokens per patch.

ModelAvg. Recon. NRMSETSAQATSETB
Qwen3-4B basen/a46.0047.3233.61
LEXIS-4B (50 symbols)0.17477.0960.7238.06
LEXIS-4B (92 symbols)0.13777.7359.6540.37

Finer quantization lowers the reconstruction error by 21%, yet TSAQA (+0.64, p=0.59p = 0.59) and TSE (−1.07, p=0.55p = 0.55) do not change significantly, and TB gains 2.31. Together with R3, this shows that 50 symbols are sufficient; we do not claim that they are optimal for every training recipe. We will add this comparison to the revision.

R5: Source of gains

Response to W5

Every row below is Qwen3-4B trained with the same recipe as LEXIS-4B, and changes one factor. Lettered shape levels, random coordinates, reading rows only and two further seeds are new runs that are not in the manuscript.

VariantControlTSAQATSETB
LEXIS-4Breference77.0960.7238.06
Preprocessing
Envelope-onlysame normalization, envelope and transient block; shape symbols removed72.1857.2433.02
Representation
Lettered shape levelssame shape levels written with existing letter tokens (no new token)75.4557.7734.69
Embedding initialization
Mean initializationsymbols initialized at the vocabulary mean (no geometry)72.7256.5736.51
Random coordinatesrandom symbol placement on the shell74.0957.6937.65
Geometric initialization, no leashgeometric placement, symbols free to move in training76.9158.9837.51
Training objective
Reading rows onlytrained only on rows whose target is text76.1859.9235.79
Noise
LEXIS-4B, mean ± sd over 3 seedstraining seed77.48 ± 0.7660.19 ± 0.4639.60 ± 1.72

We observe the following (TSAQA / TSE / TB):

  1. Preprocessing. With identical preprocessing, adding the shape symbols gives +4.91 / +3.48 / +5.04.
  2. Symbolic vocabulary. Compared with the same levels written as letters, the dedicated symbols give +1.64 / +2.95 / +3.37.
  3. Geometry. Geometric initialization gives +3.00 / +3.03 / +0.41 over random placement on the same shell and +4.37 / +4.15 / +1.55 over mean initialization.
  4. Training objective. Removing the series-writing rows costs 0.91 / 0.80 / 2.27.
  5. Noise. The standard deviation over 3 training seeds is 0.76 / 0.46 / 1.72 (R7).

R6: Unseen domains and datasets

Response to W6 and Q5

Unseen datasets. Of the three benchmarks, only TSAQA has a training split in our training data. TSE and TB are evaluated zero-shot, where LEXIS-4B improves the base LLM by 13.40 and 4.45 points. It outperforms every evaluated model on TSE, and on TB it is within 3 points of the two best models, GPT-4o (40.64) and ChatTS-14B (41.01), and ahead of all others (Table 1).

The encoder. The tokenizer is fitted once on the training corpus and not adapted. Its reconstruction NRMSE is 0.212 on TSAQA and 0.137 / 0.172 on the unseen TSE / TB, so it does not degrade outside its fit distribution.

Backbone. All trained results use Qwen3-4B. We acknowledge this as a limitation, though the tokenizer itself is independent of the backbone.

R7: Reliability of gains

Response to W7

We report paired bootstrap 95% CIs and exact McNemar tests for the comparisons of Table 1:

LEXIS −TSAQATSETB
decimals+3.36 [0.82, 5.91], p=0.01p = 0.01+2.41 [0.69, 4.13], p=0.006p = 0.006+2.91 [0.95, 4.90], p=0.005p = 0.005
soft tokens+5.36 [3.00, 7.82], p<10−4p < 10^{-4}+4.82 [1.21, 8.58], p=0.01p = 0.01+0.41 [−1.45, 2.27], p=0.70p = 0.70
envelope-only+4.91 [2.55, 7.27], p<10−4p < 10^{-4}+3.48 [0.70, 6.26], p=0.014p = 0.014+5.04 [2.16, 7.92], p=0.001p = 0.001

The lead of LEXIS is significant on all three benchmarks against decimals and envelope-only, and on TSAQA and TSE against soft tokens.

We also retrain LEXIS-4B with two further seeds:

SeedTSAQATSETB
first (reported)77.0960.7238.06
second78.3659.9239.28
third77.0059.9241.46
Mean ± sd77.48 ± 0.7660.19 ± 0.4639.60 ± 1.72

The missing tokenizer comparison is now provided in R1. We will add CIs and the mean over seeds to the tables of the revision.

On the contribution: the paper claims that a small vocabulary of symbols with known referents, installed through their geometry, is an effective interface between time series and a pretrained LLM. The new results support this claim: LEXIS matches or exceeds learned tokenizers with no tokenizer training (R1), and the gains are attributable to the symbols and their geometry (R5).

We thank the reviewer again for their feedback that strengthens this work.

Dear Reviewer FYHB,

Thank you again for the time and care you gave to our submission. We appreciate that you found our work to address an important unsolved problem with a compact and interpretable representation, and found its trained baselines, ablations and presentation useful and clear.

In our response, we added two RVQ-VAE tokenizer baselines (R1), measured how accuracy changes with reconstruction error and vocabulary size (R2 to R4), added controls that isolate each source of the gains (R5), and reported results on datasets unseen in training, confidence intervals and additional training seeds (R6, R7).

Your comments have made this work stronger, and we would be delighted to discuss any remaining questions. Please do not hesitate to contact us for any further discussion.

Authors

Comments on the paper

2 comments

The author has responded to everyone's comments. Please reply when it is convenient for you.

Dear Area Chair fCRF,

Thank you for handling our submission and for facilitating the review process. We are grateful for the valuable feedback from all three reviewers. Their comments helped us improve the work. As the discussion period comes to an end, we would like to provide a brief summary of our rebuttal.

Reviewer FYHB notes that our work addresses an important unsolved problem, representing time series effectively for LLMs, and Reviewer 6G8p finds the motivation of our approach clear. All three reviewers describe LEXIS as a compact, interpretable and decodable representation, with deterministic decoding that can be dissected and inspected clearly. Reviewer ueDQ highlights that it uses about one quarter of the tokens of decimal serialization. The reviewers also value the experimental design: the trained baselines, matched comparisons and component-wise interventions go beyond aggregate accuracy and show how the model uses the representation. Finally, all three rated the paper clear and easy to follow.

All three reviewers provided valuable feedback that substantially strengthened this work. We addressed their concerns as follows.

  • Comparison with learned tokenizers (FYHB). We trained two RVQ-VAE tokenizers with the recipe of the paper. LEXIS is competitive with both and leads them on TSAQA, with no tokenizer training and with fixed-meaning symbols, which enable a per-symbol analysis of model performance that a learned tokenizer does not allow.
  • Reconstruction fidelity and representation size (FYHB). We varied the reconstruction error in both directions around the reported model. A 92 symbol alphabet lowers the error by 21% without a significant change on TSAQA or TSE, and on TSAQA accuracy falls significantly only once 6 of the 8 shape axes are removed. The reported model therefore operates where reconstruction error does not limit understanding accuracy. We acknowledge exact point-value precision as a limitation.
  • Symbolic vocabulary against explicit temporal structure (FYHB, 6G8p). We ablate the symbolic shape levels, replacing the representation from dedicated symbols to letter tokens and found that dedicated symbols improves the model performance more.
  • Geometry against scale of the initialization (ueDQ, FYHB). We ran the radius-matched control, which places the symbols on the same shell with random coordinates. We show that meaningful placement improves the model more than random placement.
  • Connection between symbols and language (6G8p). A logit lens analysis shows that the slope symbol evokes the matching words inside the reported model (5.32 nats, absent before training), and exchanging the slope and curvature symbols drastically reduces accuracy on the relevant questions, indicating that the model reads the symbols by their meaning.
  • Supervision of the soft-token baseline (6G8p, ueDQ). LEXIS trained on exactly the examples of the soft-token baseline keeps its lead on TSAQA and TSE (+4.45 and +4.02), so this lead does not come from the additional series-writing supervision.
  • Training accounting (ueDQ). We reported the samples, optimizer steps and input tokens of every trained variant.
  • Reliability of the gains (FYHB, 6G8p). With paired bootstrap confidence intervals and exact McNemar tests, the lead of LEXIS is significant against decimals on TSAQA and TB and against soft tokens on TSAQA and TSE. We also trained two additional seeds and reported the mean and standard deviation across runs.
  • Generalization (FYHB). TSE and TB have no training split in our corpus, and LEXIS improves the base model on them by 13.40 and 4.45 points. The reconstruction error of the tokenizer on them (0.137 and 0.172) is no higher than on TSAQA (0.212).
  • Definitions in Table 6 (ueDQ). We gave the definitions and clarified that the free share is a ratio of norms, mislabeled as a share of squared norm. With the label corrected (2.2%), the reported quantities are consistent.

All new results and analyses will be added to the revision. Thank you again for your time and attention. We would be glad to provide further information in response to any follow-up questions from the reviewers.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.