acceptodds
Withdrawn from ICLR 2027

GLINT-3D: Geometry-aware Language-guided Interleaved Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial understanding from multi-view inputs requires vision-language models (VLMs) to preserve both fine-grained visual evidence and geometric relationships that can only be established across views. Feeding per-frame semantic and geometry-aware features directly to the language model is effective but computationally expensive, since the visual token sequence, and with it the decoder KV cache, grows linearly with the number of input frames. In this work we introduce GLINT-3D, a multi-view VLM built on language-guided, interleaved compression of semantic and geometric features. GLINT-3D fuses per-frame features from the VLM vision encoder with multi-view 3D-aware features from a pretrained 3D foundation model, then compresses the result into scene-level and frame-level latent tokens. Rather than compressing once before the language model, GLINT-3D initializes these latent tokens from learnable embeddings and progressively injects visual information through compression layers interleaved with the VLM layers, allowing the question tokens to guide what information is retained. Experiments on a suite of spatial and general visual benchmarks show that one-shot compression causes substantial information loss, while the combination of interleaving and language guidance largely closes the gap, approaching the mean score of the uncompressed model while substantially reducing the visual token count and decoder KV-cache size. GLINT-3D thus offers a favorable trade-off between spatial reasoning performance and scalability to larger image sets and language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
spatial intelligence, vision-language models, 3D understanding guided token compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_SohgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Combines VLM semantic features with MapAnything geometric features for multi-view spatial reasoning.

  • Introduces question-guided compressors between language-model layers to update compact scene-level and frame-level tokens.

  • Substantially reduces decoder KV-cache size while retaining much of the uncompressed model’s aggregate performance.

Critical strengths
  • Well-motivated architecture: Question-guided updates let compact tokens repeatedly access detailed visual evidence, supporting global and view-specific reasoning.

  • Informative evaluation: Multiple spatial benchmarks and compressed/uncompressed comparisons demonstrate the contribution of geometric features and expose practical trade-offs.

Critical weaknesses
  • Limited practical benefit: At 128 views, end-to-end time to first token improves by only 8%. Training peak memory increases from 31 to 70 GB, while important geometric tasks degrade substantially—for example, SPAR camera motion drops from 89.8 to 55.0.

  • Incomplete attribution of gains: A matched question-conditioned upfront compressor is missing, and the main ablations use an earlier architecture. Consequently, the benefits of interleaving and the final design remain insufficiently isolated.

Questions for authors
  • Is the upfront Perceiver baseline question-conditioned and matched in token budget and training settings?

  • Are final-model ablations available without language guidance and with token-budget-matched single-branch variants?

  • Are compressed 4B results available demonstrating accuracy, end-to-end latency, and peak memory?

  • Why does compression particularly hurt backward-facing questions when the uncompressed model uses the same training distribution?

Reviewer_1Ja4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper present GLINT-3D, An efficient pipeline for visual token compression for spatial reasoning. It use dense geometry and per-frame features as external memory and repeatedly reads them through a compact, language conditioned latent interface interleaved with its early layers. Experiments show that the method preserves most of the uncompressed model's spatial reasoning performance while reducing visual token count and KV-cache size.

Critical strengths

The paper solves an important token compression problem in scaling multiview spatial reasoning. The proposed GLINT-3D pipeline combines per-frame feature and multiview geometric feature with language guided token compression, potentially reducing the memory requirements of VLMs when processing large image sets.

The evaluations across multiple spatial reasoning benchmarks show that interleaved compression can largely preserve the performance of the uncompressed model while significantly reducing the LLM's visual token count and KV-cache size.

Critical weaknesses
  1. The paper lacks sufficient comparisons with existing token compression methods. Language guided and intermediate-layer compression have already been explored in prior works, yet the authors mainly benchmark against a basic one-shot Perceiver. I’d like to see direct comparisons with recent methods under similar compression ratios and computational budgets.

  2. The number of latent tokens is manually fixed at 200 scene-level tokens and 20 tokens per frame, without sufficient analysis of how these choices affect performance. This fixed budgets may be insufficient to represent large scale scenes or frames containing dense information. The authors should investigate adaptive token allocation and evaluate how scene scale, complexity and information density affect the accuracy.

  3. The ablations for language guidance and the scene-level and frame-level compressors are using GLINT-3D v0 not the final GLINT-3D model. Given that the final architecture introduces several structural changes, I recommend conducting these ablation experiments on the final version to ensure that the reported conclusions still hold.

  4. GLINT-3D reduces the number of visual tokens entering the LLM, but does not actually reduce the dense features produced by its encoders. Instead it relies on a multiview encoder with heavy global full attention on all the images (MapAnything). As a result, the time-to-first-token reduction is only 8% compared with the uncompressed version.

  5. Since the token compression is question dependents, this prevent us to reuse the compressed spatial feature for another question. Each question requires running the compression process again. It would be interesting to investigate whether a query-independent scene representation could be cached and combined with a smaller query-specific retrieval component, avoiding repeated compression.

Questions for authors
  1. How does GLINT-3D compare with recent query guided and intermediate layer compression methods under the same token budget?
  2. Can the authors repeat the main component ablations using the final architecture?
  3. How sensitive is the method to the fixed allocation of 200 scene tokens and 20 tokens per frame? Can the authors provide ablations across different token budgets, scene scales, and information densities?

Although this paper tackles a meaningful problem in multi-view token compression with promising initial results. I find there are multiple concerns among the evaluations and ablation studies, As a result, I won't give a positive score at this moment. I suggest the authors to further revise the paper and solve my concerns above.

Reviewer_vivjOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GLINT-3D, a multi-view VLM for spatial reasoning. It fuses per-frame features from the VLM vision encoder with multi-layer geometry-aware features from a frozen 3D foundation model (MapAnything) through an AdaLN-Zero merger. The fused features are then compressed into 200 scene-level latents and 20 latents per frame.

The compression does not happen once before the LLM. The latents start as learnable embeddings inside the LLM input sequence. They are updated by k=6 compressor layers inserted every m=2 LLM layers. From the third compressor layer onward, each layer also cross-attends to the hidden states of the question tokens (“language guidance”). Weights are shared between the scene and frame compressors and across depths, leaving two weight sets: with and without language guidance.

Training has two stages. Stage 1 is captioning-heavy alignment with a frozen LLM. Stage 2 is spatial QA plus VSI-style instruction tuning with LoRA. The data mixture includes in-house templated QA from ScanNet, ScanNet++, ARKitScenes, 3RScan and MultiScan.

Evaluation covers VSI-Bench, VSI-Bench-Debiased, ReVSI, SPAR-Bench and CV-Bench. The final model reaches a mean of 68.7, compared with 70.4 for the uncompressed variant (GLINT-3D-NC) and 58.4 for an up-front Perceiver compressor. The authors estimate a 6–9× reduction in KV-cache size.

Critical strengths
  1. The problem is relevant. Token and KV-cache growth with the number of views is a real obstacle for multi-view spatial reasoning, especially once 3D foundation-model features are added on top of semantic features.

  2. The core idea is simple and sensible. Persistent latents act as a working memory that repeatedly reads dense features, conditioned on the evolving question representation. This is a natural alternative to compressing everything before the model has seen the question.

  3. Broad empirical evaluation and informative ablations. The experiments cover several complementary benchmarks, including VSI-Bench, ReVSI, SPAR-Bench, and CV-Bench. The authors consider both 2D-only and geometry-aware variants, compressed and uncompressed models, and a one-shot Perceiver baseline. The improvement from the one-shot compressor to the interleaved design provides encouraging evidence for the proposed approach.

I also appreciate that the paper reports training memory, wall-clock time, and inference latency rather than relying exclusively on theoretical token reduction.

Critical weaknesses
  1. The ablation studies do not fully support the contributions of interleaving and language guidance.

The paper presents interleaving and language guidance as its two main contributions. However, the only one-shot compression baseline in Table 1 is a Perceiver compressor without language guidance. It is therefore unclear how much of the improvement comes from interleaving itself, rather than simply making the compression question-aware.

Also, the main ablations in Table 1(a) are conducted on GLINT-3D v0. The final model changes several important parts of the architecture, including the compressor placement, injection frequency, and language guidance schedule. The authors assume that the conclusions from v0 still hold, but this is not directly verified.

The current results show that the proposed method outperforms a standard Perceiver compressor, but they do not fully establish the contribution of each key design choice in the final model.

  1. The practical efficiency gains are limited, and scalability to larger LLMs has not been demonstrated.

Table 2 reports a 6–9× reduction in KV-cache size. However, this only measures the LLM's KV cache, rather than the memory cost of the full system.

According to Section 5.1, GLINT-3D reduces end-to-end time to first token by only 8% at 128 frames, while inference peak memory and decoding speed remain almost unchanged. During training, the compressed model even uses more peak GPU memory than the uncompressed version (70 GB vs. 31 GB). This suggests that reducing the LLM's visual tokens does not necessarily lead to a large efficiency gain for the full system, especially when MapAnything dominates the computation.

In addition, all completed experiments with the compressed model use a 0.8B LLM. The reported 4B results are from the uncompressed model, so they do not demonstrate how the proposed compression method performs with larger LLMs.

Overall, the results support the reduction in LLM KV-cache size, but the claims about end-to-end efficiency and scalability to larger language models need stronger evidence.

  1. Compression causes large performance drops on some important cross-view spatial reasoning tasks.

The paper aims to preserve geometric information while compressing visual tokens. However, some results show substantial performance drops on tasks that require reasoning across views.

On SPAR-Bench, CamMotion drops from 89.8 to 55.0, and ViewChange drops from 54.5 to 27.6 after compression. On ReVSI, Relative Direction drops from 80.5 to 65.1. These are important tasks for the multi-view spatial understanding problem that the paper aims to address.

The authors partly explain the ReVSI results by pointing to backward-direction questions that are missing from the training data. However, the uncompressed model uses the same training setup and performs much better. This explanation alone does not account for the additional performance loss caused by compression.

The small gap in average scores may therefore hide substantial losses in specific spatial reasoning abilities. I think these failures deserve more analysis.

Questions for authors

Q1. On the contributions of interleaving and language guidance (Weakness 1)

How would a one-shot, question-conditioned compressor perform under the same visual features, latent token budget, and training setup?

Also, could the authors provide an ablation without language guidance on the final GLINT-3D model? These two comparisons would help clarify the separate contributions of interleaving and language guidance, without requiring a complete rerun of all ablations.

Q2. On practical efficiency and scalability (Weakness 2)

Could the authors clarify the memory usage of the full GLINT-3D system, especially the cost of keeping the dense visual features (X) during compression? Is this cost included in the reported peak GPU memory?

Given the relatively small end-to-end speedup, under what model sizes or input lengths do the authors expect GLINT-3D to provide a clear practical advantage?

For larger LLMs, if results with trained compressed models are not available, it would be helpful to clearly distinguish the expected KV-cache savings from the efficiency gains that have actually been measured.

Q3. On the loss of geometric information (Weakness 3)

Could the authors further explain the large performance drops on CamMotion, ViewChange, and ReVSI Relative Direction?

In particular, how do the frame-level latent tokens represent frame identity, ordering, and viewpoint information? An analysis of a few failure cases, comparing the compressed and uncompressed models, would help clarify what information is lost during compression and why these tasks are particularly affected.

Reviewer_LPYqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to handle long multi-view inputs in VLM without letting the visual token sequence and KV-cache grow too much. The authors propose GLINT-3D, which combines semantic features from Qwen3.5 with geometry features from MapAnything. GLINT-3D compresses the visual tokens into scene-level and frame-level latent tokens. The compression is interleaved with the early layers of the LLM, and the question tokens are used to guide what visual information should be kept.

The experiments show that a simple perceiver compression loses a noticeable amount of spatial information, while the proposed interleaved and language-guided design recovers much of this gap. On several spatial benchmarks, GLINT-3D performs better than the fine-tuned Qwen3.5 baseline and stays relatively close to the uncompressed 3D model, while reducing the KV-cache by roughly 6–9× for long inputs. Overall, the paper presents a reasonable combination of 3D feature fusion and query-adaptive visual compression for multi-view spatial reasoning.

Critical strengths
  1. Effective visual token compression with limited performance loss. GLINT-3D reduces the visual context and KV-cache by roughly 6–9× for long multi-view inputs, while remaining close to the uncompressed model on most spatial benchmarks. The performance degradation is relatively small compared with the compression ratio, which makes the approach practically interesting for long visual sequences.

  2. A spatially meaningful latent design. The separation between scene-level and frame-level latent tokens is a sensible inductive bias for spatial intelligence. Scene tokens can preserve global layout and geometry, while frame tokens retain view-specific evidence. Together with query-guided and interleaved compression, this design is better matched to multi-view spatial reasoning than a simple one-shot visual resampler.

Critical weaknesses
  1. The end-to-end efficiency improvement is less convincing than the KV-cache reduction. The paper reports a large reduction in KV-cache size, but does not provide TFLOPs or a comprehensive full-response latency and throughput analysis. Since MapAnything dominates the computation, the reported end-to-end time-to-first-token improvement is only about 8% at 128 frames. Thus, the method demonstrates efficient LLM-side context compression more clearly than complete system-level acceleration.

  2. The comparison with recent token compression methods is incomplete. The paper mentions recent methods such as LVPruning, DUET-VLM, and QMoP in the Related Work section, but does not compare GLINT-3D with them under a matched backbone, training data, frame sampling strategy, and token budget. As a result, it remains unclear whether the proposed compression mechanism is superior to these alternatives for multi-view spatial reasoning.

  3. The evidence for scalability and robustness is still limited. Most compressed experiments use the 0.8B model, while a full compressed 4B result is not provided. In addition, compression causes noticeable degradation on some fine-grained geometric tasks, such as relative-direction reasoning in ReVSI and medium-level tasks in SPAR-Bench. These results suggest that the method is promising, but its behavior with larger models and more demanding spatial evidence remains unclear.

Questions for authors
  1. Can the authors provide a more complete end-to-end efficiency analysis, including TFLOPs, full-response latency, throughput, and a breakdown of the costs from MapAnything, the merger, the compressors, and the LLM?

  2. Can the authors compare GLINT-3D with some of the recent token compression methods under a matched backbone, training data, frame sampling strategy, and token budget?

  3. Can the authors provide results for a fully compressed larger model, such as GLINT-3D-4B, and analyze whether the degradation on fine-grained geometric tasks persists at larger scales?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.