acceptodds
Withdrawn from ICLR 2027

GeoPACT: Geometry-Preserving Token Compression for Spatial Reasoning in VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models excel at semantic recognition and visual question answering, yet remain unreliable at spatial understanding: eg. metric distance, object size, relative direction, and multi-view scene geometry. This limitation becomes critical for embodied agents, where visual context grows continuously while memory and latency budgets remain constrained. Although geometry-aware encoders provide rich spatial representations, their dense patch features are costly to process and poorly aligned with language-model embeddings. Compressing them aggressively is therefore essential—but conventional language-only training can discard precisely the local and metric information needed for spatial reasoning. In this paper, we introduce a general training scheme that explicitly preserves spatial information during compression. During projector pretraining, lightweight auxiliary heads reconstruct dense segmentation and depth targets from the compressed tokens, encouraging them to retain object-level and geometric cues. The heads are removed before downstream fine-tuning, adding no inference cost. We validate this approach with two complementary encoders: DUNE, which provides fine-grained semantic features, and MapAnything, which captures metric multiview geometry. We then introduce GeoPACT, a lightweight dual-encoder VLM that compresses these representations into 20 tokens per view and 200 scene-level tokens—only 840 visual tokens for 32 views—and couples them with a 0.8B language model. Experiments on 2D and 3D spatial-reasoning benchmarks show consistent improvements, while ablations and frozen-token probes confirm that segmentation and depth information remain more accessible after compression. GeoPACT demonstrates that strong visual compression can preserve geometry, enabling scalable spatial reasoning for embodied VLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, spatial reasoning, 3D scene understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_M3YROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GeoPACT, a framework design that explicitly preserves spatial information during compression for VLMs. By leveraging two off-the-shelf encoder modules and training a connector, GeoPACT achieves high-ratio compression (i.e. 20 tokens per view). Experimental results show that GeoPACT achieves improvements on various benchmarks including 2D and 3D.

Critical strengths
  1. The research question is correct and important. How to compress multi-view images into compact token sets for VLMs is a long-standing and hard problem, so research in this branch has a solid background.
  2. The idea of Auxiliary Supervision is interesting. Attaching lightweight dense-prediction heads to Zb and training on pseudo-labels is an interesting way to ensure feature effectiveness. This is a method worth attention in the field of 3D VLMs.
Critical weaknesses
  1. Limited novelty. The main technical contribution, adding a connector merging features from various vision encoders is not novel. The selection of visual encoders and the design of connectors are well-explored in “mixture of vision experts” literature: Cambrian-1 [1], Eagle [2], MoVA [3]. Therefore, I don’t actually think either of the contributions listed at the end of Introduction holds.
  2. Weak performance and experimental design. It is surprising to see GeoPACT, which undergoes two stages of training, underperform Qwen3.5-0.8B-FT by ~10 percentage points in 2 of VSI-Bench and all categories of CV-Bench 3D. This undermines the effectiveness of the proposed method, and raises doubts for whether the performance gain in other categories is merely due to a distribution shift compared with Qwen3.5 model. Besides, it is not convincing to conduct experiments on a 0.8B model, which can barely be called a large model.

[1] Tong, Shengbang, et al. "Cambrian-1: A fully open, vision-centric exploration of multimodal llms." Advances in Neural Information Processing Systems 37 (2024): 87310-87356. [2] Shi, Min, et al. "Eagle: Exploring the design space for multimodal llms with mixture of encoders." International Conference on Learning Representations. Vol. 2025. 2025. [3] Zong, Zhuofan, et al. "Mova: Adapting mixture of vision experts to multimodal context." Advances in Neural Information Processing Systems 37 (2024): 103305-103333.

Questions for authors

Due to my recognition of this paper as of limited novelty and weak experimental performance, I would personally recommend authors reconsider what is their contribution, sharpen their claims and observations, redesign their method to gain noticeable performance gain and conduct experiments on larger models, before submitting to another conference.

Reviewer_7S6hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GeoPACT, a spatial VLM that drops the native vision encoder of Qwen3.5-0.8B and replaces it with frozen DUNE and MapAnything. DUNE features are compressed to 20 tokens per view, and the MapAnything scene representation to 200 tokens, giving 840 visual tokens for 32 views. These tokens are projected into the language-model embedding space and concatenated with text tokens. In Stage 1, segmentation and depth heads are attached to the compressed tokens; the targets come from the encoders' own decoders, and the heads are removed before Stage 2. Stage 2 fine-tunes the LLM with LoRA on a mixture that includes VSI-590K and the 340K VLM-3R video questions. Evaluation covers VSI-Bench debiased, ReVSI, CV-Bench, and SPAR, all through VLMEvalKit with 32 frames.

Critical strengths

The ablations in Table 1 are run under one training recipe. DUNE alone is stronger on single-image questions, and MapAnything alone on multi-view ones; using both gives 62.1 on VSI-Bench debiased, 48.4 on ReVSI, 72.8 on CV-Bench 2D, and 70.2 on CV-Bench 3D—above the stronger single branch on every benchmark. The auxiliary heads are ablated in the same recipe: without them, VSI is 57.0 and CV-Bench 2D is 51.3; with hard segmentation and depth pseudo-labels, 62.1 and 72.8, with either head alone also improving over the no-head row. Within this model, the dense pseudo-labels do make the compressed representation more useful.

Critical weaknesses

1.Under matched data, the architecture does not show a clear gain over the original vision encoder, and the comparison with VLM-3R is confounded by training data. Qwen3.5-0.8B-FT uses the same language model and the same Stage-2 mixture; its five-benchmark average is 63.1 against 63.7 for GeoPACT. GeoPACT is 2.6 higher on VSI and 5.0 higher on SPAR, 0.4 lower on ReVSI, and 70.2 against 79.1 on CV-Bench 3D, with route planning and appearance order also lower. VLM-3R's 53.3 on VSI in Table 3 comes from the official checkpoint, while GeoPACT's training mixture already includes VLM-3R's 340K questions. The re-evaluated RoboBrain score of 33.9 on VSI, and SPAR scores around 42 for both 7B models, sit below the untuned Qwen3.5-0.8B at 58.1. I would not treat these gaps as evidence for the architecture until VLM-3R is retrained on the same data. 2.The evidence for the auxiliary heads is also narrower than the abstract, and stops one step short of the claim. The heads supervise the compressed tokens before the language model, so "geometry is preserved" is enforced by the training objective rather than demonstrated; Figure 2 reconstructs from a head trained on those very tokens, which shows the information is decodable at the interface, not that it survives anywhere else. Nothing in the paper tests whether geometric information is present in, or used by, the LLM itself—there is no probe on LLM hidden states, and no analysis distinguishing "the tokens carry geometry and the LLM sometimes reads it" from "the LLM's internal representation remains geometry-agnostic and the gains come from token content alone." Combined with the soft-target degradation (VSI 54.3, below the no-head 57.0), the causal chain from supervision to spatial reasoning gains is asserted rather than shown.

Questions for authors

1.Could you report VLM-3R retrained on the same Stage-2 mixture under the same 32-frame protocol? If GeoPACT still clearly exceeds it on VSI-Bench debiased and ReVSI, the architectural claim would be substantially better supported, and I would raise my rating. 2.Why is the auxiliary supervision placed at the compression interface rather than deeper in the network (e.g., on LLM intermediate layers, as in OLA-VLM)? Was a deeper placement tried and found worse? I ask because supervising at the interface only guarantees that the input tokens retain geometry; it does not show that geometric capability is transferred into the LLM. A linear probe on LLM hidden states for metric quantities (depth, coordinates), compared between with- and without-head models, would directly test this. 3.GAP-MLLM also uses geometric supervision so that language training does not wash out 3D information. What is the key difference between your approach and theirs? A quantitative comparison on a shared benchmark would be helpful.

Reviewer_Xm6GOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoPACT, a compact vision-language model and a training scheme designed to preserve spatial information during visual token compression. The model combines two frozen encoders: DUNE provides fine-grained per-view features, while MapAnything captures metric geometry across multiple views. Separate Perceiver resamplers compress these features into a small set of image tokens and a fixed set of scene tokens, which are supplied to a 0.8B language backbone. The main contribution is to supervise these compressed representations directly during projector pretraining. Lightweight auxiliary heads reconstruct dense segmentation and depth targets generated by the encoders’ existing perception decoders, encouraging the resamplers to retain object boundaries, depth relationships, and metric information alongside language-relevant features. Each branch is initially pretrained independently against a frozen language model. The auxiliary heads are then removed, and both branches are jointly fine-tuned with LoRA adaptation of the language model. This allows the auxiliary supervision to shape the compressed representation without adding inference-time components.

Critical strengths
  1. The controlled auxiliary-supervision study provides strong evidence for the proposed training scheme. With the architecture and token budget held fixed, segmentation and depth supervision improve performance across the evaluated benchmarks, both for the dual-encoder model and for individual encoders. These results support the value of supervising compressed representations beyond the benefit of simply combining two visual branches.
  2. Applying auxiliary supervision directly to the tokens consumed by the language model offers a coherent way to address information loss during compression. The training procedure clearly separates independent branch pretraining from joint instruction fine-tuning, with auxiliary heads removed before the latter stage. The benefits therefore require no additional auxiliary tokens or prediction heads at deployment, giving the approach practical value despite its use of established components.
  3. The encoder ablations support the complementary roles of global scene geometry and localized visual information. MapAnything performs better on multi-view benchmarks despite its smaller token budget, while DUNE is stronger on single-image tasks. Combining them improves over either branch alone across the reported evaluations. Although total capacity is not fully controlled, these findings support the practical motivation for the dual-encoder design.
Critical weaknesses
  1. The explanation of geometry preservation is less well supported than the downstream improvements. The paper repeatedly refers to frozen-token probes, but their quantitative results and evaluation protocol are not clearly presented. Qualitative reconstructions, attention maps, and probes of uncompressed encoder features do not establish that geometry remains more accessible in the compressed tokens, particularly after Stage 2 updates the projectors without auxiliary supervision.
  2. Two central aspects of the auxiliary objective remain underspecified. The paper does not explain how the scene prediction head distinguishes target views when reconstructing different maps from the same global tokens. It also omits the depth discretization or distribution construction needed for cross-entropy and KL supervision. These details are essential for reproducing the method and interpreting the hard-versus-soft comparison.
  3. The separation between training and evaluation scenes is unclear. The custom supervision draws from datasets that also contribute to ReVSI, but scene-level exclusions and overlap checks are not documented. Shared source datasets do not establish contamination; however, an explicit scene manifest or intersection analysis is needed to support claims of generalization to unseen scenes.
  4. The novelty discussion needs a more precise comparison with reconstruction and distillation methods. OLA-VLM already uses pretraining-only predictors and projector learning through a frozen LLM, while ROSS receives little mechanism-level discussion.
  5. The aggregate VSI scores do not match the macro-averages of the corresponding task-level results. Given the public evaluator’s category-level aggregation, the paper should clarify whether different checkpoints, evaluator versions, or aggregation rules explain this discrepancy. The inconsistency does not negate the reported gains, but makes their magnitude difficult to audit.
  6. Claims of compactness and scalability should distinguish the language backbone from the complete deployed system. Only the scene-token budget is fixed; per-view tokens still grow with the input length, and visual encoding dominates runtime. The comparison with the fine-tuned Qwen baseline shows similar latency and average performance, alongside weaker CV-Bench 3D results. Broader efficiency claims should reflect these trade-offs, and the conflicting precision settings should be reconciled.
Questions for authors
  1. Could you identify the frozen-token probing experiments referenced in the abstract, Section 4.1, and conclusion, and provide their protocol and quantitative results? The most informative comparison would use matched probe capacity and held-out data for models trained with and without auxiliary supervision. Please also specify whether the representations are taken after Stage 1 or Stage 2, since the latter continues updating the compression modules after removing the auxiliary heads.
  2. How is the target view specified to the scene-level auxiliary head when all views share the same 200 scene tokens? Please describe any view embeddings, camera conditioning, view-specific queries, or other conditioning used. In addition, how are MapAnything’s metric-depth predictions converted into hard labels and soft distributions, including the depth range, discretization, invalid-target handling, and loss normalization?
  3. Which part of the contribution do you consider distinct from the already-cited OLA-VLM and ROSS? In particular, do any existing results distinguish the value of the proposed depth/segmentation targets from a generic reconstruction objective at the same compression budget? A focused comparison or a narrower statement of the contribution would be more informative than adding many unmatched system baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.