acceptodds
Under review as a conference paper at ICLR 2027

TokAudit: Auditing What Extra Capacity Buys in Time-Series Tokenizers

Abstract

Tokenizer capacity is usually treated as a scalar budget, but reconstruction error does not reveal what extra capacity preserves. We introduce TokAudit, a measurement framework for auditing marginal capacity increases by separating reconstruction gains into latent-structure and observation-noise components, benchmarking their allocation against a matched reference, and measuring downstream utility independently. Across the tested tokenizer constructions, most of what extra capacity buys in reconstruction is observation noise rather than latent structure, while learned codecs are usually less noise-oriented than the matched reference. More noise-oriented allocation is also associated with smaller downstream utility gains, but this relationship is population-dependent. TokAudit therefore separates capacity growth, reconstruction composition, and task utility, reframing tokenizer scaling as an auditable allocation problem.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.