- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Lexical Lens separates representation analysis into a reference metric derived from the input and output lexical interfaces, decoder-free prediction of terminal states, and a three-coefficient convex forward-KL calibration. The paper argues that this construction permits coordinate-consistent measurement of semantic states and transitions. Across ten language models and 322 layers, it reports macro held-out KL of 2.033 versus 2.984 for Tuned Lens, with most of the improvement already obtained by the decoder-free predictor.
Critical strengths
-
Useful separation of geometry, prediction, and calibration. The balancing objective for the raw positive-definite lexical Grams has a unique matrix-geometric-mean solution with the stated covariance law. Once the response fields and affine decoder are frozen, the three-coefficient KL objective is convex and contains the decoder-free prediction as a feasible anchor. These are substantive mathematical components with clear roles. [Sections 3.1–3.3; Appendix A.6]
-
Broad and carefully grouped predictive evaluation. The experiments span ten models, use disjoint calibration and test sequences, and select the response scale by excluding complete sequences. Improvements are reported on all 322 layers and 640 held-out sequences, with secondary metrics and calibration-budget analyses. The decoder-free ablation supports the practical value of the terminal-state estimator. [Section 4; Appendices A.2 and A.5; Tables 2 and 6]
Critical weaknesses
-
The stated implementation does not establish covariance of the complete estimator. States are represented as with orthogonal , while the metric is stored separately. Consequently, cancels from the ridge response in Eq. (3), leaving an identity penalty in native coordinates; using the metric only to select the scale does not repair this. For and , matching the original response after transformation requires both and , which are incompatible for , , and . Appendix A.3 also describes an identity-array Riesz lift despite the nonidentity metric, while Appendix A.1's unit-determinant normalization removes the metric scaling required under . Undocumented reciprocal coordinates could resolve some discrepancies, but the necessary conventions are absent. Table 1 tests incorrect shortcuts rather than covariance of the refitted full estimator. [Eq. (3), p. 4; Appendices A.1–A.3, p. 13; A.5–A.6, p. 15]
-
The predictive comparisons do not isolate the contribution of the consensus reference. The proposed shrinkage estimator is compared with an Adam-trained affine translator, without a native-coordinate state-ridge baseline using the same sequence-excluded selection or controls using individual lexical interfaces. In addition, Tuned Lens applies final normalization after its translator, whereas the proposed convex fit uses affine logits in the predicted state. These differences leave regularization and output normalization as alternative explanations for the KL improvement. The results support predictive utility more directly than the claimed necessity of the consensus geometry. [Sections 4.1–4.3; Appendix A.5]
Questions for authors
-
What exact coordinate transformations implement the ridge operator, diagonal backbone, and Riesz lift? Please reconcile the identity-lift statement and unit-determinant normalization with the metric law, and establish whether this requires clarifying existing code or changing the estimator.
-
Does refitting the implemented pipeline commute with dilation, determinant-one anisotropic scaling, and shear? Resolving the first concern requires an explicit derivation addressing the counterexample above and numerical agreement of transported held-out predictions and semantic transitions, including any necessary refits.
-
How much improvement remains with matched state-ridge, single-interface metric, and output-normalization controls? A higher rating would require these controls to substantiate the consensus reference's contribution alongside a consistent implementation of the claimed geometry.
4 comments
W2 / Q3: Predictive controls and the role of Consensus Geometry.
State-ridge control. Across all ten models, Lexical Lens reduced held-out KL relative to native ridge by 0.13%–4.35% before Information Projection and 0.81%–5.07% after it. When both fits selected shrinkage by Euclidean State error, the balanced-axis backbone still lowered Final KL by 0.75%–5.00% versus plain ridge on every model. These controls use separate fits on the models and evaluation data of Table 2.
Both fits use the native ridge penalty and select its strength by leaving out whole sequences. Native ridge predicts the terminal State directly and scores Euclidean errors. Lexical Lens fits a slope and intercept per balanced axis, applies ridge to the residual, and scores Consensus-weighted errors. Each fit receives its own final KL projection on the same data. Entries are layer-mean held-out KL in nats (322 layers total).
| Model | State: Lexical Lens* | State: ridge | Final: Lexical Lens | Final: ridge |
|---|---|---|---|---|
| SmolLM2 135M | 1.897 | 1.900 | 1.742 | 1.756 |
| SmolLM2 360M | 2.010 | 2.029 | 1.801 | 1.822 |
| SmolLM2 1.7B | 2.277 | 2.302 | 1.982 | 2.020 |
| MiniCPM5 1B | 1.698 | 1.707 | 1.579 | 1.596 |
| MiniCPM5 2B | 2.142 | 2.156 | 1.945 | 1.978 |
| Granite 4.2 3B | 2.794 | 2.845 | 2.319 | 2.386 |
| Granite 4.2 8B | 3.212 | 3.301 | 2.695 | 2.806 |
| Ministral 3 3B | 2.328 | 2.368 | 2.020 | 2.083 |
| Ministral 3 8B | 2.479 | 2.540 | 2.153 | 2.242 |
| Ministral 3 14B | 2.396 | 2.505 | 2.092 | 2.204 |
All 20 paired 95% intervals favor the submitted fit over plain ridge (held-out sequences resampled with fits fixed). Under Euclidean scale selection, the native-axis backbone lowered mean Final KL by 0.73%–4.95% relative to plain ridge, accounting for most of the balanced-axis backbone's gain on each model. Changing only to balanced axes further lowered mean Final KL on nine of ten, with all changes below 0.30% in magnitude.
Single-interface control. Across all ten models and 322 layers, we changed only the metric for scoring excluded sequences, keeping the balanced axes, native penalty and gradient lift fixed. Negative changes in KL favor the alternative.
| State-risk metric | Smallest Final-KL change | Largest Final-KL change |
|---|---|---|
| Identity | −0.222% | +0.205% |
| Input metric | −0.355% | +0.052% |
| Output metric | −0.592% | +0.701% |
Changing the scale-selection metric alone moved Final KL by less than 0.71% in either direction. On four models (118 layers), we also rescaled the jointly balanced axes to use one interface for the penalty, lift and scale selection together. Final KL rose on all four, by 0.004–0.152 nats for the input metric and 0.005–0.057 for the output metric.
Output-normalization control. The Lexical Lens readout stayed affine. Switching on the translator's output normalization lowered KL in all eight pairs with raw or normalized inputs on four models (118 layers). Only output normalization varied within each pair. The translator used 56 sequences to fit and eight to validate. Lexical Lens used all 64. Both used the same 64 held-out sequences.
| Held-out KL | SmolLM2 360M | MiniCPM5 1B | Granite 4.2 3B | Ministral 3 3B |
|---|---|---|---|---|
| Raw source, output norm on | 2.586 | 2.402 | 3.172 | 2.676 |
| Raw source, output norm off | 71.696 | 12.893 | 13.694 | 2.927 |
| Normalized source, output norm on | 2.595 | 2.402 | 3.141 | 2.679 |
| Normalized source, output norm off | 3.589 | 4.902 | 7.614 | 4.691 |
With normalized inputs and output normalization enabled, translator KL was still 0.66–0.82 nats higher than Final Lexical Lens. Removing normalization to match the affine readout widened that gap on all four models.
Role of the Consensus reference. The input metric measures a State displacement relative to the spread of token embeddings. The output metric measures its effect on centered decoder logits. Each raw metric is covariant. Consensus supplies one ruler across depths by minimizing the larger of its affine-invariant distances to the two interfaces, before any predictor is fitted. Appendix A.6 gives the midpoint proof and Table 7 the measurements.
Balancing gives , with unique positive-definite solution . For determinant-normalized interfaces a distance apart, mean-square distortion is at Consensus and at either interface alone.
| Models | Identity excess distortion over Consensus |
|---|---|
| 4 models with tied embeddings | 6.08%–15.90% |
| 6 models with untied embeddings | 22.36%–96.92% |
Using only frozen weights, we found greater distortion for native identity on every model. In the submitted fit, joint balancing sets the backbone axes, Consensus weights State errors for scale selection, and native normalization supplies the ridge penalty and gradient lift. The revision gives the per-model distances and distortions.
Thank you for working through the two-axis example. Appendix A.6 in the revised PDF clarifies which metric supplies the ridge penalty and gradient lift and how it transforms, which resolves my covariance concern under the stated assumptions. The numerical refits under dilation, anisotropy, shear, and rotation also confirm that the predicted states and transitions agree in practice. Those refit tables and the predictive control tables are currently only in your comments rather than the PDF, so it would help to include them alongside the proof in the appendix.
The matched-ridge and single-interface controls also clarified what Consensus Geometry contributes to prediction. While the full predictor beats matched ridge on all ten models (and Table 8 shows it remains ahead of the Muon-trained Tuned Lens), your breakdown shows that with Euclidean scale selection, the native-axis backbone already accounts for nearly all of the gain over plain ridge (0.73% to 4.95%). Switching to the balanced Consensus axes changes final KL by less than 0.30% per model, and changing only the State-risk metric to the identity moves final KL by less than 0.25%. The geometric motivation for a common reference is clear, but its actual contribution to predictive performance looks quite small here, so I am keeping my rating at 2.
The remaining disagreement concerns the role of Consensus Geometry within Lexical Lens. The submitted abstract describes it as fixing a “data-free, model-wide reference,” and the introduction explicitly separates reference, dynamics, and readout. Before any layerwise predictor is fitted, the two frozen token interfaces determine one set of fitting axes and one metric for scoring State errors across all depths. Observable Dynamics then estimates what each source layer predicts about the terminal State. The submitted abstract attributed most of the gain over Tuned Lens to this decoder-free prediction stage.
Table 7 shows why the choice of reference matters. Relative to Consensus, choosing either interface alone doubles the mean-square distortion to the pair, and using the native Euclidean metric increases it by 6.08%–96.92% across the ten models. With this common reference, successive State predictions read through the same frozen decoder show how predictable support for each token changes between layers. Your follow-up confirms that the covariance concern is resolved under the stated assumptions. Across all ten models and four tested coordinate changes, complete refits reproduced these transitions. The maximum layerwise relative error was below 0.066% when held-out positions were pooled within each layer.
Both ablations retain a metric and fitting axes shared across layers. Each model’s mean final KL changed by less than 0.25% when only the shrinkage-selection metric was replaced by the Euclidean metric, and by less than 0.30% when only the axes were changed under Euclidean scale selection. These figures show that final KL is relatively insensitive to those substitutions. The separate matched-ridge control directly fitted the terminal State in native coordinates. With each fit given its own final projection, Lexical Lens achieved 0.81%–5.07% lower held-out KL on all ten models. The two ablations alone are therefore insufficient grounds for retaining a 2 for the complete method. We ask you to reconsider the rating and the Area Chair to weigh both the predictive comparison and the verified cross-layer measurements in the meta-review.
Thank you for your detailed review, especially the counterexample you provided and the experimental controls you suggested. We address your concerns point by point below.
W1 / Q1: Exact coordinate transformations and the counterexample.
You are right that U cancels from the native ridge response. Your two-axis example shows why retuning one scalar cannot repair a penalty reset to identity. The submitted c=xU fit uses native normalization for the ridge penalty and gradient lift, and the balanced token interfaces for the backbone axes and State-risk metric. Our added code carries these choices into a new chart to refit the same estimator after normalized State extraction (Sections 3.1–3.3 and Appendix A.6).
Under x′=xT, the native normalization metric becomes N′=T−1NT−⊤ and the penalty is ρ(N′)−1. In your X=I2 example, ρI becomes ρT⊤T, so T(T⊤T+ρT⊤T)−1T⊤=(1+ρ)−1I. The original ρ now recovers the response on both axes.
The Riesz lift uses the native normalization metric, so its matrix is the identity in native and orthogonal execution coordinates where N=I. For a native decoder covector Γ, the lifted State direction in the new chart is (ΓT−⊤)(N′)−1=(ΓN−1)T.
The backbone fits one slope per balanced axis, giving a generally non-diagonal map in native coordinates. If A balances the unit-determinant interfaces, c=xA(A⊤N−1A)−1/2 reduces to xU in the native chart. Rebuilding the axes gives c′=cS and B′=BS for the same diagonal sign matrix S. The raw Consensus metric follows the law for N. Fixing its determinant multiplies the transported metric by ∣detT∣2/d. The square root of this scalar cancels in the normalized State-risk factor, so dilation preserves every candidate's score.
Transporting the decoder preserves logits and KL for every final coefficient triple. Corresponding optima give x′=xT, identical token readings and identical transitions. This assumes exact arithmetic, positive-definite interfaces and a distinct, consistently ordered balanced spectrum. Repeated eigenspaces and nonunique optima require consistent choices.
W1 / Q2: Complete refits under dilation, anisotropic scaling and shear.
Independent refits under dilation, anisotropic scaling, shear and rotation agreed with the native predictions on the original ten models (322 layers). Pooling held-out positions at each layer, the maximum relative error was below 0.019% for Final predictions and 0.066% for Semantic Transitions. These refits supplement the shortcut tests in Table 1.
In each chart we rebuilt the axes from the transformed interfaces and N′ and refitted the backbone, ridge scale and projection. State and Final denote native-coordinate predictions before and after Information Projection. Semantic Transitions compare adjacent-depth token readings before projection. Entries are maximum relative Frobenius errors in fractions. At a single held-out position, the largest relative Transition error was 0.451%.
The largest Final discrepancy with transport occurred under anisotropic scaling at layer 12 of Granite 4.2 8B. Against the same native fit and at the same precision, the relative error was 11.42% when we reset the metric to identity and 0.0182% when we carried it. Resetting the penalty changes the relative shrinkage along the two axes in your example. With the metrics carried and the axes rebuilt, fitting before or after the coordinate change recovered the same held-out final State and depthwise change in token readings to the accuracy shown above.