- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits Jacobian-lens measurements used to characterize a verbalizable workspace. It examines depth signatures, an operational onset rule, and occupancy estimators across open-weight model–lens artifacts. It finds strong dependence on model, locator, and unembedding-centering convention, and distinguishes an effective-dimensionality transition from a functionally established workspace boundary.
Critical strengths
- The reference-free checks, corpus-sensitivity analysis, and planted-count tests identify concrete measurement dependencies rather than interpreting every numerical difference as a model property.
- The paper carefully separates supported and unsupported conclusions. In particular, the centering controls and the causal patching probe prevent the spectral onset from being presented as a universally established functional boundary.
Critical weaknesses
- The broad cross-family onset range pools artifacts with unequal fit sizes, provenance, and validation coverage; the two largest self-fitted lenses lack retained gate artifacts. The range is descriptive rather than an isolation of family effects, and the incomplete validation limits the strength of that central cross-artifact result.
- The functional evidence remains narrow: the capital-city patching test uses one model and one intervention site, and its onset is later than the spectral transition. This supports the paper's caution, but leaves the positive scientific meaning of the surviving onset measure limited.
Questions for authors
- How does the cross-artifact range change under a common fit size and complete reference-free validation on a matched subset?
- Does the gap between spectral and functional onset persist when intervention sites include earlier token positions or another relation?
Thank you for your patience. Both checks are done.
We ran a second relation, "X is a city in the country of," and it lands in the same place as the capitals one: onset at 80.0% of depth, same 20-point gap. We tried a third, "the currency of X is," and couldn't make it work. The model's top answer is " the" on all 69 prompts, so there's nothing for the probe to measure. That's our template rather than the model, and we report it that way.
On earlier positions, patching at the subject tokens transfers from 0% of depth and at all positions from 8%, both well before the onset, which inverts what we'd predicted. The whole-state controls explain why. The answer moves from the subject tokens to the final position between 68% and 72% of depth, and that is exactly where swapping the whole state at the final position starts working. So the 20-point gap is roughly 12 points before the answer has even arrived at the last position, and 8 after it's there but before the J-space part is what carries it. One model, one relation, and we don't claim more than that.
On fit size, your question turned up something in our paper. The Gemma-2-9B lens was fitted at SEQ=40 while the paper states SEQ=128. Refitting at the stated setting moves the onset from 61.0% to 65.9%. The two curves correlate at 0.9998, so it’s a threshold crossing rather than a different result, and the 19.7-point spread doesn’t change. Both numbers are in the revision.
On the two largest fits, we have them now. They were on a co-author’s storage rather than lost, and the paper was wrong to say otherwise. Gemma-2-27B passes the identity check. Qwen3-32B doesn’t, and neither has a positive control, since that needs weights we don’t hold.
The 32B failure isn’t ours alone. The published checkpoint fails the same way at a different precision and prompt count, and the two curves agree at 0.993. So the check doesn’t hold for that architecture either way. Dropping Qwen3-32B would make our spread 13.3 points rather than 19.7, so we report both numbers and keep the model in. On the matched subset itself, the three fits that carry both per-lens gates span 6.7 points, though those three are small models from two families, so validation quality, family, and scale aren’t separable there.
The revision is up. Thank you again for your time!