Language models (LMs) predict the next-token probability distribution over a large vocabulary, yet most of the probability mass is often concentrated on a small, context-dependent subset. This raises the question of whether LM outputs across contexts admit a compact representation. To answer this question, we first characterize this behavior of LM outputs using Bhattacharyya geometry on probability simplexes and simplicial complexes from algebraic topology. Our theorem connects probability mass discarded by truncation to our geometric dissimilarity measure. Because each output point lies near the low-dimensional simplex of its high-probability tokens, the output points are covered by a neighborhood of a simplicial complex, a union of simplexes. Next, we evaluate representations of LM outputs through the tradeoff among four metrics: (d,m,r,u). The first two measure the complexity of simplicial complexes, while the latter two assess approximation quality. To construct compact covers, we first consider collecting pointwise covers. For a single output, Top-k and Top-p truncation determine optimal simplexes in the tradeoff between r and d with fixed m=1 and u=0. Collecting these simplexes yields simplicial complexes, which contain a prohibitively large number of simplexes in practice. To obtain a more compact complex, we formulate the construction task as a monotone submodular maximization problem and propose an algorithm for this problem. The algorithm provides the same worst-case guarantee as previous work. In the experimental section, we evaluate the construction using a natural language corpus (Dolma) and real-world LLMs (OLMo, Pythia). One of the resulting simplicial complexes uses 843 simplexes of dimension at most 4096 to cover 95% of the evaluated outputs while retaining at least 90% of each covered output's probability mass, providing a compact representation of approximately 50K-dimensional LM outputs over a 100M-token corpus.
Thank you for asking whether the implementation correction affects coverage and how our covers compare with random covers of comparable size. Your questions helped us show how much coverage changed after the correction and how our method compares with random shared covers under the same size limits.
1. Regarding results by corrected implementation
Short response: We reran the experiments with corrected implementations and confirmed that the results remain largely unchanged.
Your concern about the mismatch between the guarantee and the implementation is justified. Repeating the experiments with the corrected procedure is necessary to determine whether the reported findings still hold.
We corrected the threshold initialization and repeated the evaluation over the same parameter grid for OLMo and Pythia. The minimum target threshold is now strictly positive, and the initial supports are always included in the candidate stream.
The results remain largely unchanged: based on the reported values, coverage changes by no more than 0.1 percentage points in 49 of 54 settings. In the remaining five settings, coverage improves by 1.2–2.2 points for OLMo and 3.3–3.7 points for Pythia. No setting shows a decline in coverage at this precision.
The corrected evaluation therefore preserves the empirical findings and addresses the concern that they depend on the flawed implementation. We will revise the figures and implementation description accordingly.
2. Regarding comparison with randomly constructed simplicial complexes
Short response: Yes, the simplicial complexes constructed by our method outperforms randomly constructed supports.
We agree that comparisons with pointwise Top-k/Top-p constructions alone do not establish the benefit of selecting shared supports. This requires comparison with simple shared covers under comparable budgets.
Following your suggestion, we constructed baseline simplicial complexes from 1) randomly selected admissible Top-p supports, 2) supports sampled from a unigram-weighted distribution, and 3) supports sampled uniformly over the vocabulary. These constructions used the same nominal dimension and simplex-count limits. We then evaluated all covers on held-out Dolma documents that had not been used to construct any of these complexes.
Our covers outperform the random Top-p baseline with the same construction threshold in all 81 comparisons per model (27 settings and three evaluation thresholds). Table 1 compares the methods at the central grid setting for OLMo, where our cover achieves the highest coverage among the listed baselines.
These comparisons provide evidence that coverage-based selection using Sieve-Streaming improves on simple shared-cover constructions. The observed benefit therefore goes beyond the pointwise concentration captured by Top-k/Top-p.
Table 1. OLMo coverage at a representative parameter.
*Single Top-p and single uniform: column-wise medians over 16 draws.
Your questions helped us notice which comparisons were missing, and the results of those comparisons now lend more direct support to our conclusions.