acceptodds
Under review as a conference paper at ICLR 2027

How Should VLMs Represent Ego-State? A Diagnostic Study on Representation Fidelity and Context Cost

Abstract

Vision-language models increasingly receive numerical state, such as vehicle kinematics, alongside images, yet the trade-off between how accurately and how compactly this state can be delivered is rarely measured. We measure it for four interfaces that receive the same ego-state history in Qwen3-VL-2B: decimal text, LLMTime-style text, xVal-style scalar embeddings and learned sixteen-position adapters. All share downstream supervision, train on CARLA and are evaluated on held-out CARLA and real nuScenes clips. Decimal text is the most accurate at . Our staged adapter, pretrained on reconstruction and captions before task fine-tuning, reaches (three-run mean ) while shortening the state segment and the full input , and joint continuation raises it to . Beyond aggregate accuracy, the two sixteen-position recipes fail on different question types, and three staged runs within 1.63 points of each other reach recall from to on 175 nearly stationary clips, which explain most of the difference between the weakest and strongest run. State interfaces should therefore be chosen by accuracy and context cost and evaluated per task across runs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal representation learning, vision-language models, continuous vs. discrete representations, time-series ingestion, multimodal adapters, context efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wTHqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the trade-off between task accuracy and context cost when feeding ego-vehicle kinematic histories into a vision-language model, primarily Qwen3-VL-2B-Instruct with LoRA. Then it measures four interfaces that receive the same ego-state history: decimal text, LLMTime-style text, xVal-style scalar embeddings and learned sixteen-position adapters.

The main finding is that accuracy drops monotonically as the state representation is compressed, and the paper appends over 30 pages of post-hoc tables and subgroup breakdowns to support the idea.

Critical strengths
  1. Useful core question and baseline comparison: It is a practical and useful baseline for embodied VLMs to measure the accuracy-vs-budget trade-off of text serialization against continuous embeddings on a shared multivariate state history.
Critical weaknesses
  1. Auto-research presentation with post-hoc excuses instead of a research paper: As noted in the AI Use Statement, generative AI was used across ideation, code revision, analysis suggestion, diagram drafting, and writing. In practice, the 43-page manuscript reads like an uncurated automated-research log: 31 pages of appendices and 41 tables mechanically dump numbers accompanied by repetitive, formulaic disclaimers while providing very little genuine scientific insight.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer wTHq

Dear Reviewer,

Thank you for recognizing the practical relevance of our research question. We were shocked that our transparent AI-use statement and the appendix’s length and presentation could lead to the impression of autonomous research without substantive author direction and curation. We strongly disagree with that characterization and would like to explain our research process.

This was author-led research. The authors designed the study, made the methodological decisions, selected and interpreted the analyses, and determined the conclusions. We reviewed AI-assisted writing and code, checked the reported results and citations, and take responsibility for the submission.

1. What our AI-use statement describes

We reported assistance wherever it occurred, including activities beyond language polishing, to describe our workflow transparently.

Writing and figure preparation. The authors developed the storyline. For some sections, an LLM drafted paragraphs based on that structure, which we rewrote and refined. Figure 1 was produced through Python code written with LLM assistance. The authors directed its concept, layout, and revisions, and checked the numerical values and reported results.

Literature discovery. We used AI-assisted search, including Gemini functionality in Google, and related-work suggestions from LLM feedback. The authors checked the citations included in the submission.

Methodological discussion and analyses. We used LLM feedback, including simulated reviews, to discuss methodological choices and identify questions for investigation, similar in purpose to the optional automated author-feedback programmes offered at NeurIPS 2026 and ICLR 2027. The authors assessed that feedback, decided which analyses to pursue, and interpreted the evidence. We disclosed this as research ideation and execution because the feedback influenced scientific decisions.

Code assistance. Coding agents assisted with parts of the project, particularly analysis and plotting code. The authors reviewed the generated code and resulting plots.

Synthetic caption generation. Our study investigates VLMs, where model-generated training supervision is established practice, as illustrated by LLaVA (Liu et al., 2023) and BLIP (Li et al., 2022). Qwen3-8B generated the wording of Stage-2 alignment captions from deterministic motion facts (§3.3). We disclosed this training-data construction step.

2. Relationship to ICLR’s policies

The ICLR AI Policy for Authors requires disclosure of assistance with hypotheses, experimental methodology, method implementation, interpretation, and synthetic-data generation. It recommends disclosure of drafting, editing, literature discovery, figures, and other code assistance. We reported the relevant activities across both groups. The policy also invites detailed accounts: “we encourage you to provide as much detail as you want!”

3. Purpose and presentation of the appendix

Our study investigates how trained VLMs use numerical state. Given the complexity of these learned interfaces, we used additional empirical analyses to examine the scope of the main findings and aimed for a sufficient documentation of limitations and failure modes. The main paper presents the central argument and principal evidence. The appendix provides further experiments and methodological detail.

The opening guide, “Where each claim is supported,” within the appendix maps claims to the relevant sections. Introductions such as “This section supports Section …” recur because readers were expected to consult individual sections through that guide. We therefore repeated those connections without prioritizing variation in their wording and provided an overview table at the beginning of the appendix (similar to a glossar).

We acknowledge that we concentrated on conducting experiments and interpreting results and did not revise the appendix’s language sufficiently for a full read of the appendix rather than a glossary lookup. Some passages are consequently repetitive and insufficiently polished. We accept responsibility and will streamline the repeated framing and improve the explanations of each analysis’s purpose and findings.

AuthorsOpenReview ↗

Response to Reviewer wTHq (2/2)

4. Clarification of the quoted examples

The review describes every appendix section as ending with repetitive automated disclaimers. The examples it cites require the following clarifications:

Example cited in the reviewManuscript wording, context, and purpose
“Ranks are descriptive”This phrase recurs in table captions to distinguish observed rankings from statistical significance claims. We repeated it so that readers consulting individual tables would have the qualification available alongside the results.
“These comparisons condition on the checkpoints and do not estimate retraining variability”This exact sentence occurs once, in the introductory paragraph of §E.2 (p. 22), not at the subsection’s end. It states the uncertainty scope defined in §3.4 (p. 6): intervals conditional on evaluated checkpoints do not estimate variability under retraining. Variations of this qualification are used to keep the scope of individual comparisons explicit.
“This does not establish a common mechanism”The manuscript states, “This does not identify a common mechanism,” within §E.2 (p. 22), before the magnitude-shift paragraph. It cautions against inferring a shared causal mechanism from a shared performance pattern.

We accept the criticism that the appendix needs further language revision. We reject the implication that this shortcoming reflects research conducted without substantive author direction, scientific judgment, or verification. We apreciate the positivity about our general idea and would be open to discuss the scientific content.

I appreciate the experimental effort and the inclusion of controls. However, the reason why I flagged the paper a fully auto research paper is that I find the paper reads more like an experimental coverage report than a research paper with clear motivation, interpretation, and actionable conclusions.

  1. The reason behind the comparison needs clearer framing.

    The interfaces have origins, but the paper does not sufficiently explain why these particular methods lead to the appropriate comparison or what hypothesis each tests.

  2. The practical value of the accuracy–cost trade-off remains unclear. The adapter reduces state-segment length by 48.8×, but the full input reduction is 2.16× and the measured prefill speedup is 1.54×, with noticeable accuracy decreases from 96.00% to 88.57%.

  3. The experiments reveal differences without sufficiently explaining their causes. Interfaces differ in architecture and optimization, and the three staged runs differ in preparation length as well as seed. These comparisons therefore do not isolate whether the observed failures arise from representation capacity, alignment, training, or other factors.

  4. The recommendations largely restate the measured trade-off. Section 4.5 recommends text for accuracy, adapters for context savings, and xVal as an intermediate option. This offers limited guidance beyond the reported rankings.

  5. The organization emphasizes experimental execution over the central argument. Numerous recipes, stages, and diagnostics make the paper difficult to follow.

Overall, I do believe the paper contains useful measurements, but it needs a stronger connection between the application need, experimental design, and resulting guidance. At present, I will keep my clear rejection recommendation.

Reviewer_rsmuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to represent numerical ego-state histories for VLMs, comparing decimal text, LLMTime-style text, xVal-style embeddings, and learned adapters under shared downstream supervision. It proposes a compact adapter trained through state reconstruction, caption alignment, and task adaptation. Experiments on EgoDyn-Bench, using CARLA training data and held-out CARLA and nuScenes clips, show that trained decimal text achieves the highest accuracy (96.00%), while the staged adapter reaches 88.57% with a 48.8× shorter state segment and 2.16× shorter full input. Additional diagnostics reveal task-specific errors and substantial subgroup performance variation across adapter training runs, motivating evaluation beyond aggregate accuracy.

Critical strengths
  • Overall, this paper is well written and easy to follow.
  • The motivation is clear and reasonable.
  • Some experiments do demonstrate the effectiveness of the proposed method.
Critical weaknesses
  • The practical accuracy–efficiency trade-off is not sufficiently established. The staged adapter reduces state-context length substantially, but increases benchmark error from 4.00% with trained decimal text to 11.43% (2.86×), and joint continuation still leaves 10.05% error. Meanwhile, the reported 1.54× speedup applies only to prefill with cached vision features, rather than end-to-end inference. The adapter also adds approximately 105M parameters and requires reconstruction and caption-alignment stages. These costs make it important to establish when the compact interface is preferable to text.

  • The main interface comparison lacks validation across backbones and tasks. The ranking of decimal text, LLMTime-style text, xVal-style embeddings, and learned adapters is established only on Qwen3-VL-2B. The InternVL experiments compare two adapter encoders but omit the corresponding trained text and scalar-embedding baselines, so they do not validate the central interface ranking or its accuracy–cost trade-off. Matched interface baselines on the second backbone, together with broader task validation, are needed to substantiate that guidance.

Questions for authors

Please refer to weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer rsmu

Dear Reviewer rsmu,

Thank you for your positive feedback regarding our writing, clear motivation, and experimental validation, as well as your helpful critique of the efficiency-accuracy trade-off and backbone validation.

1. Practical Accuracy–Efficiency Trade-Off

We agree that the trade-off between compactness and accuracy must be rigorously quantified. Below is the direct comparison of performance, context length, and computational cost across the evaluated interfaces:

InterfaceBenchmark accuracyMean full-input positionsCached-vision prefill
Decimal text96.00%1,42556.6 ms
xVal-style embeddings92.17%91339.1 ms
Staged adapter88.57%66136.8 ms

While the adapter provides a minimal context footprint and faster prefill (reducing the full input length by 2.16× and state-segment positions by 48.8×), it incurs an accuracy cost (88.57% vs. 96.00% for text, with joint continuation raising it to 89.95%) and adds ~105M parameters plus reconstruction and caption-alignment preparation stages. As noted in §4.5, these compact interfaces become critical when scaling to longer time horizons and multi-agent scenes where text inputs consume excessive context. We also emphasize that prefill time savings do not translate to equivalent end-to-end inference speedups.

2. Interface Validation and Backbone Scope

Regarding the lack of matched trained text and xVal baselines on the second backbone, we clarify that the complete interface comparison and ranking are specific to Qwen3-VL-2B on EgoDyn-Bench.

The InternVL experiments were not intended to replicate the full 11-row baseline table, but rather to serve as a rigorous cross-architecture diagnostic. Specifically, InternVL tests whether the low-speed stationary clip failures and encoder ranking shifts (where the xLSTM-inspired encoder outperforms the Transformer) generalize beyond Qwen. We will make this distinction explicit in the main text.

This rebuttal fails to address my core concern: the practical accuracy-efficiency trade-off of the proposed adapter remains unestablished. This is an autonomous driving task where state estimation error is a primary, safety-relevant metric. The 2.86× error increase (from 4.00% to 11.43%) is a severe and unacceptable degradation. The reported speedup only applies to prefill with cached vision features, with very limited prefilling speedup.

Reviewer_8QaqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to feed a vehicle's state history, such as position and velocity, into a vision-language model. The authors compare text, numerical embeddings, and a short-sequence adapter, and train an adapter that compresses the state history into 16 input positions. Experiments show that text achieves higher question-answering accuracy, while the adapter can save context and prefill cost. Further analysis finds that models with similar overall accuracy can differ drastically on the same set of low-speed clips. Overall, this is a careful empirical comparative study that reveals local failures which aggregate scores tend to hide.

Critical strengths
  1. The research question is practically meaningful. The vehicle state occupies the model's context, and the input format also affects answering performance. The authors measure both accuracy and actual cost at the same time, providing a concrete reference for choosing an input format.
  2. The analysis of low-speed clips is noteworthy. Across three training runs the overall accuracy differs by less than two percentage points, yet the accuracy on the same low-speed clips ranges from 8% to 98.3%. This shows that models with similar overall performance may have very different capability weaknesses.
Critical weaknesses
  1. The justification for adopting the new method is still insufficient. The adapter saves context and GPU memory, but its question-answering accuracy is lower than xVal. In the prefill test with cached visual features, the time is only shortened by about 2 milliseconds. The current results do not explain when these resource savings are worth the corresponding accuracy cost.
  2. The analysis mostly stops at describing the phenomenon. The authors find that different models differ drastically on low-speed clips, but do not identify the key cause of the failures.
Questions for authors

None

2 comments

Dear Reviewer,

Thank you for your constructive feedback regarding the justification of the adapter and the analysis of low-speed failures. Below, we address your concerns with detailed empirical results from our study.

1. Investigation and Cause of Low-Speed Failures

To investigate why models with similar overall accuracy diverge sharply on low-speed clips, we fitted a multinomial logistic regression readout directly to the 16 flattened projected state tokens (after standardization and 512-component PCA, using training data only).

On all 1,000 peak-half questions, comparing this linear readout against the VLM’s original multiple-choice scoring yields the following breakdown:

Correct classExamplesReadout recallVLM recall
First half34896.6%96.8%
No peak27887.8%36.0%
Second half37496.0%95.7%
Overall accuracy1,00093.9%79.5%

This discrepancy also appears in seed 43 and two full-fine-tuning checkpoints. Conversely, seed 44 reverses the ordering, yielding 94.2% VLM accuracy versus 92.5% readout accuracy.

These findings demonstrate that the projected state tokens successfully preserve task-relevant information, isolating the performance drop to how the VLM's multiple-choice scoring utilizes those tokens across training runs rather than an information bottleneck in the adapter itself.

2. Accuracy–Efficiency Trade-Off and Resource Justification

To evaluate whether the resource savings justify the accuracy cost, we measure performance, context length, and prefill cost side-by-side:

InterfaceBenchmark accuracyMean full-input positionsCached-vision prefill
Decimal text96.00%1,42556.6 ms
xVal-style embeddings92.17%91339.1 ms
Staged adapter88.57%66136.8 ms

The staged adapter reduces the full input length by 2.16× (and the state segment by 48.8×) while achieving 36.8 ms prefill compared to 56.6 ms for decimal text. Joint continuation raises adapter accuracy to 89.95%.

These efficiency gains come at an accuracy cost (88.57% vs. 96.00% for text) and require ~105M additional parameters plus preparation stages. As discussed in §4.5, these resource savings are designed for scenarios where context constraints are tight, such as long horizons or multi-agent scenes, where text-based representations scale linearly and quickly saturate VLM context windows. Furthermore, the prefill measurements do not imply equivalent speedups for end-to-end inference.

I thank the authors for the response and the additional analysis.

The linear-readout experiment addresses my second concern directly. Separating "the information is absent from the tokens" from "the information is present but not used by the multiple-choice scoring" is the right diagnostic, and the result is informative.

My first concern remains. The response restates the measured trade-off rather than establishing when the resource savings justify the accuracy cost. The scenarios invoked as justification — long horizons and multi-agent scenes — are not evaluated in the paper, and the reported speedup applies to prefill with cached visual features rather than end-to-end inference.

I therefore maintain my rating of 2 (weak rejection).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Reviewer Response

Dear Reviewers,

Thank you for your thoughtful and constructive engagement with our work. We are encouraged that both reviewers recognize the practical meaningfulness of our research question and the value of our diagnostic findings—particularly the identification of hidden subgroup failures in low-speed clips that aggregate metrics typically conceal.

To summarize the core contribution and anchor the discussion across reviews, our paper provides the first controlled comparative study of numerical-state interfaces for vision-language models under a strictly unified setup: holding the exact same multivariate ego-state history, backbone (Qwen3-VL-2B), and downstream supervision constant.

The central trade-off established across our evaluation is summarized below:

InterfaceBenchmark accuracyMean full-input positionsCached-vision prefill
Decimal text96.00%1,42556.6 ms
xVal-style embeddings92.17%91339.1 ms
Staged adapter88.57%66136.8 ms

Key Takeaways & Strengths of the Study:

Clear Design Guidance: Text achieves the highest absolute accuracy (96.00%), while our staged adapter achieves a 2.16× reduction in full-input length (and a 48.8× reduction in state-segment length) with faster prefill (36.8 ms), which joint continuation further raises to 89.95%. This provides concrete reference points for choosing interfaces based on strict context and latency constraints.

Beyond Aggregate Scores: Our diagnostics reveal that models with near-identical overall accuracy can diverge drastically on specific subsets (e.g., nearly stationary low-speed clips), proving that learned interfaces require per-task evaluation across training runs rather than relying solely on aggregate scores.

Below, we address each reviewer’s specific technical questions.

Dear reviewers,

As the Author–Reviewer discussion period concludes in a few days, we kindly urge you to read the authors’ rebuttal and respond as soon as possible.

Please review all author responses and other reviews carefully, and engage in open and constructive dialogue with the authors.

The authors have addressed comments from all reviewers; each reviewer is therefore expected to respond, so the authors know their rebuttal has been considered.

We strongly encourage you to post your initial response promptly to allow time for meaningful back-and-forth discussion.

Thank you for your collaboration, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.