How Should VLMs Represent Ego-State? A Diagnostic Study on Representation Fidelity and Context Cost

Vision-language models increasingly receive numerical state, such as vehicle kinematics, alongside images, yet the trade-off between how accurately and how compactly this state can be delivered is rarely measured. We measure it for four interfaces that receive the same ego-state history in Qwen3-VL-2B: decimal text, LLMTime-style text, xVal-style scalar embeddings and learned sixteen-position adapters. All share downstream supervision, train on CARLA and are evaluated on held-out CARLA and real nuScenes clips. Decimal text is the most accurate at $96.00\%$. Our staged adapter, pretrained on reconstruction and captions before task fine-tuning, reaches $88.57\%$ (three-run mean $89.30\%$) while shortening the state segment $48.8\times$ and the full input $2.16\times$, and joint continuation raises it to $89.95\%$. Beyond aggregate accuracy, the two sixteen-position recipes fail on different question types, and three staged runs within 1.63 points of each other reach recall from $8.0\%$ to $98.3\%$ on 175 nearly stationary clips, which explain most of the difference between the weakest and strongest run. State interfaces should therefore be chosen by accuracy and context cost and evaluated per task across runs.