Context Is King: How In-Context Specification Shapes the Geometry of Concepts

Large language models place structured concepts on geometrically faithful manifolds: weekdays lie on a circle, months on another, often read as a fixed world-model the network stores and looks up. Prior work shows that context can affect entity-token geometry while a strong pretrained prior remains prominent. We show where context takes over: at the final prompt token, after the model has read the rule and query, geometry follows the in-context specification and dominates the prior. A declarative rule changes not just the ordering of entities but the geometry's relational topology: the same tokens form a cycle or a hierarchy with tree depth and branch relationships on command. This change cannot be explained by simple relabeling and holds even for arbitrary nouns randomly assigned to the structure's nodes. Under conflicting specifications, capable models in both tested families (Gemma, Qwen) show representational similarity of $0.6$–$0.9$ to the imposed structure versus near-zero to the pretrained order, across the concept sets tested. To investigate the computation underlying these geometric findings, we use activation patching and find that replacing one entity's activation with another redirects the answer to the donor's successor under the context-specified order. Sweeping the intervention across layers localizes where entity identity enters this computation. Smaller models can already form rough context-specified geometry, while larger models more consistently suppress the pretrained order and follow the active relation under patching. This family-dependent trend cautions against transferring mechanistic conclusions across model sizes. Whether the model builds this geometry anew or reconfigures a stored one remains open; operationally, the geometry it uses is the one the context specifies. Code and artifacts: https://github.com/anonauthoriclr/context-is-king.