acceptodds
Withdrawn from ICLR 2027

Scalable Feature Coding for Distributed Large Vision-Language Model Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Feature coding enables distributed inference of large vision-language models (LVLMs) by compressing visual representations at the edge and transmitting them to the cloud for downstream processing. However, using a single coded representation for tasks with different information requirements can waste bandwidth and incur unnecessary computation in the cloud-side large language model (LLM). To address these problems, we propose the first Scalable Feature Coding (SFC) framework for LVLMs. SFC organizes visual representations into a hierarchical bitstream, comprising a base layer that encodes coarse-grained global semantics and two enhancement layers that provide additional information for object-level and pixel-level semantics, respectively. This hierarchy enables coarse-grained tasks to operate on fewer visual tokens while supplying finer-grained information on demand. To further improve coding efficiency, we analyze correlations across and within information layers and introduce two complementary context models. Cross-layer context modeling (CL-CTX) conditions the entropy model of each enhancement layer on previously decoded layers, exploiting inter-layer dependencies for more accurate probability estimation. Intra-layer context modeling (IL-CTX) further reduces coding redundancy by capturing dependencies within a layer. Experiments with InternVL on image captioning, object detection, and instance segmentation demonstrate that SFC achieves 18.37% – 51.30% BD-BR and 7.5% – 55.6% end-to-end LVLM inference latency over baseline methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
feature coding, coding for machines, large vision language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YvMSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Scalable Feature Coding (SFC) for distributed large vision-language model inference. Visual tokens are ordered using a diversity-based traversal and partitioned into a base layer and enhancement layers. Different cumulative token budgets serve image captioning, object detection, and instance segmentation. Layer-specific learned codecs use hyperpriors, cross-layer attention-based entropy context, and two-pass intra-layer context. Experiments use an InternVL2.5-based model and compare rate–accuracy performance against feature-coding baselines, along with latency comparisons against image-transmission pipelines. Additional experiments evaluate semantic segmentation and visual grounding.

Critical strengths

(1) The framework offers explicit cumulative decoding access points. Earlier reconstructed latents are available before enhancement-layer decoding, and the two-pass intra-layer context avoids fully sequential autoregressive decoding. This provides a coherent implementation of progressively accessible visual features.

(2) The paper evaluates both task accuracy and compression efficiency. Table 3 provides useful controlled evidence that cross-layer and intra-layer contexts improve the proposed codec, while Appendix A.4 explicitly discusses position-signaling overhead. These details strengthen the technical credibility of the implementation.

Critical weaknesses

(1) The experiments do not isolate the benefit of scalable coding from token reduction and stronger entropy modeling.

The proposed system changes several factors simultaneously: the number of retained tokens, their ordering and partitioning, the number of separately parameterized codec branches, and the entropy-context architecture. Figures 5 and 8 demonstrate improvements for the complete system, but do not establish which gains specifically require a shared scalable bitstream.

A particularly important missing control is to apply the same token selection and cumulative budgets to the existing feature codecs. Without this comparison, the captioning and detection gains may substantially reflect encoding fewer tokens rather than the proposed hierarchy. TOFC already combines token merging with learned feature coding, so its inclusion alone does not establish a matched-token comparison; the adaptation and token-budget settings need to be specified explicitly.

The paper also lacks independently optimized, non-scalable codecs for the same access points. Such a comparison would quantify the rate–accuracy cost or benefit of supporting several prefixes within one bitstream. The context ablations in Table 3 demonstrate improvements within the layered design, but do not answer whether this design is preferable to selecting an appropriately sized non-scalable representation when the task is already known.

This distinction matters because the reported experiments evaluate each task separately, whereas a distinctive benefit of scalable coding is reuse of previously transmitted information when additional detail is subsequently requested. The paper does not evaluate this progressive-request setting or quantify its advantage over task-selected independent representations.

Furthermore, the fixed allocation of tasks to token budgets is motivated by a small set of task-level sensitivity curves. The ordering algorithm optimizes geometric diversity, and Appendix A.1 appropriately acknowledges that its guarantee does not establish task accuracy or semantic disentanglement. Consequently, the interpretation of the layers as global, object-level, and pixel-level semantics is stronger than the evidence for a fixed token-budget hierarchy.

Overall, the paper establishes a promising combined system, but does not sufficiently establish the distinctive contribution of scalability over simpler alternatives built from its component techniques.

(2) The latency evaluation is insufficient to substantiate the claimed distributed-inference advantage over relevant alternatives.

Figure 7 compares SFC against centralized JPEG/BPG pipelines. These comparisons change both representation coding and computation placement: the centralized baseline performs visual encoding in the cloud, while SFC performs visual processing and feature coding at the edge. The result is a valid comparison between complete pipelines, but it does not isolate the latency benefit of the proposed codec or establish superiority over the distributed feature-coding baselines in Figure 5.

Table 2 likewise compares reduced-token SFC against the original full-token model. It establishes a cloud-side benefit from input compaction, but does not show that scalable coding is necessary to obtain that benefit. A matched-token baseline could obtain similar prefill and KV-cache reductions.

The operating points used for Figure 7 are also insufficiently specified. The manuscript does not clearly identify the exact rate–accuracy points selected for each method or establish that their task accuracies are matched. Consequently, the reported latency reductions cannot be fully assessed as equal-quality comparisons.

Appendix A.3 states that all experiments use RTX-3090 GPUs, and the transmission analysis uses three low-bandwidth settings. This supports a particular GPU-based, bandwidth-constrained operating regime. It leaves the applicability to resource-limited edge deployment uncertain, especially because the edge must execute the visual encoder, diversity ordering, and learned feature encoder. The ordering includes sequential selection rounds, whose contribution should be measured explicitly.

A complete timing protocol should identify whether the reported latency includes token ordering, all codec transforms, actual entropy encoding and decoding, position signaling, cloud prefill, and full output generation. These details are especially important when comparing tasks with substantially different output lengths and when transmission becomes less dominant.

The current results therefore demonstrate potential latency savings against selected image-transmission pipelines, but provide insufficient evidence for the broader efficiency advantage claimed over relevant distributed feature-coding alternatives.

Questions for authors

(1) Do the rate–accuracy improvements remain when LaMoFC, DT-UFC, and TOFC use the same token ordering and the same cumulative token budgets as SFC? Please specify the token counts, training data, adaptation procedure, and model capacity used for each baseline.

(2) How does SFC compare with independently trained non-scalable codecs at each access point under matched token budgets and comparable codec capacity? What is the cost of supporting multiple prefixes, and what benefit appears when a second task requests enhancement information after the base layer has already been transmitted?

(3) How sensitive are the conclusions to the fixed layer allocations? Were these allocations selected using validation results, and do alternative allocations or simpler ordering methods achieve comparable performance under the same budgets?

(4) Which exact operating points are used in Figure 7, and are task accuracies matched across methods? What are the corresponding payload sizes and complete latency breakdowns?

(5) Does the timing include visual encoding, diversity ordering, learned transforms, actual entropy coding, position signaling, prefill, and complete output generation? How do the feature-coding baselines perform under the same hardware, bandwidth, and equal-accuracy protocol?

(6) In Table 1, does “unseen tasks during training” refer specifically to codec training? Please clarify the task exposure of the underlying Tang et al. checkpoint and distinguish codec transfer from generalization to tasks unseen by the LVLM itself.

Reviewer_k7esOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Scalable Feature Coding, a hierarchical compression framework intended to reduce communication costs and cloud-side inference computation for distributed large vision-language models. SFC partitions visual tokens into a base layer and enhancement layers using diversity-guided token ordering. Cross-layer context modeling and intra-layer context modeling exploit dependencies between and within layers to improve entropy coding. Experiments cover three primary tasks, with additional evaluation on semantic segmentation and visual grounding.

Critical strengths
  1. The paper discusses a practically important problem that different vision-language tasks require different levels of visual information, yet conventional feature coding typically transmits the same complete features regardless of task. Its progressive feature decoding framework has potential to reduce communication costs and inference latency in edge-cloud collaborative inference and low-bandwidth settings.

  2. Quantitative evidence show that the reduced representation can lower cloud-side inference costs, rather than improving compression rate alone.

Critical weaknesses
  1. Limited methodological novelty. The proposed cross-layer and intra-layer context models appear to be combinations or adaptations of existing context modeling techniques for learned feature compression. The paper would benefit from a clearer explanation of the fundamental methodological differences from prior work and a more compelling demonstration of its novel contributions.

  2. Generalization ability remains underexplored. The experiments primarily focus on a single vision-language model and features derived from COCO. It remains unclear whether the proposed framework can generalize to different model architectures, visual encoders, and broader task distributions.

Questions for authors
  1. Insufficient evidence for semantic hierarchy. The proposed farthest-first token ordering is based on feature-space diversity, but geometric diversity does not necessarily correspond to semantic importance or task-specific information. The paper does not provide sufficient evidence that the resulting feature layers consistently capture coarse-to-fine semantic information.

  2. Relation to Learned Image Compression. The paper does not sufficiently discuss the connections and differences between SFC and existing learned image compression methods[1,2,3]. From my understanding, they both compress features of images for compact latent representations. A more thorough comparison with relevant approaches would help clarify the technical novelty of the proposed context modeling and scalable coding design.

  3. Higher-Bitrate Robustness. The evaluated bitrate range appears to be concentrated around 0.1–0.5 BPFP. It would be valuable to extend the rate–accuracy evaluation to higher-bitrate settings and verify whether SFC maintains consistent and scalable gains over existing baselines.

  4. Scalability Design Choices. The paper evaluates only two- and three-layer configurations, but does not sufficiently justify these heuristic choices or analyze how the number of layers affects the rate–distortion trade-off. Experiments with additional layer configurations or a systematic analysis of the trade-offs would help establish whether the chosen configurations are well motivated and how scalable the proposed framework is in practice.

  5. Unclear ablation studies. The evaluation lacks comparisons with alternative token-ordering strategies, making it difficult to isolate the benefits of the proposed diversity-guided ordering. In addition, more comprehensive experiments are needed to disentangle the contributions of token ordering, progressive coding, and the two context models.

  6. Edge Deployment Evaluation. The paper claims the importance of deployment of cloud-side large language model but actually does not evaluate SFC on resource-constrained edge devices or investigate its sensitivity to different hardware capabilities. Direct deployment results on representative edge hardware are needed to substantiate its practical feasibility and clarify the trade-offs between compression efficiency and edge-side computational overhead.

[1] Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. CVPR 2022.

[2] Learned Image Compression with Mixed Transformer-CNN Architectures. CVPR 2023.

[3] Mambaic: State space models for high-performance learned image compression. CVPR 2026.

Reviewer_berPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Scalable Feature Coding (SFC) for distributed large vision-language model inference. It organizes visual tokens into a base layer and enhancement layers, allowing tasks with different information requirements to use different portions of a shared bitstream. Cross-layer and intra-layer context models improve coding efficiency. The method addresses both communication overhead and cloud-side computation, giving it a clear practical motivation. However, the experiments do not sufficiently establish its effectiveness on larger, more recent models or its end-to-end latency benefits on unseen tasks.

Critical strengths
  1. The motivation is clear and practically relevant. Transmitting information according to task requirements offers a way to reduce both communication overhead and the number of visual tokens processed by the cloud-side LLM.

  2. The layered representation and context models form a coherent design. The feature-correlation analysis motivates CL-CTX and IL-CTX, and the ablations in Table 3 support their contributions to coding efficiency.

Critical weaknesses
  1. Limited evaluation on larger, more recent models. The experiments focus on InternVL2.5 and do not demonstrate how the method extends to larger, more recent LVLMs. It is unclear whether the coding and latency gains persist when the backbone, feature dimensions and inference costs change. The authors should specify the exact model size and include experiments on a larger model, such as Qwen3-VL-32B [1].

  2. End-to-end latency benefits on unseen tasks have not been established. Table 1 and Figure 8 evaluate semantic segmentation and visual grounding, but report rate–accuracy results without corresponding end-to-end latency measurements. It therefore remains unclear whether the transmission-time savings outweigh token-selection and codec overhead on unseen tasks. Please report the full pipeline latency, including selection, encoding, transmission, decoding and LLM inference, at comparable task performance.

  3. The comparisons do not isolate the contribution of scalable coding. The effects of token selection, token count and codec capacity are not sufficiently separated from those of layered coding. Controlled comparisons should use identical token selections and counts, with matched codec capacity, to compare scalable and non-scalable coding. Both rate–accuracy performance and latency should be reported.

[1] Shuai Bai et al. Qwen3-VL Technical Report. 2025.

Questions for authors
  1. The visual inputs evaluated in the paper are limited to still images. Can SFC be extended to video understanding, and how would the information layers handle temporal information? This would help assess whether the method generalizes beyond the image-only setting studied here.

  2. For a new task, how does SFC determine which information matters, how much to transmit and which layers to decode? Are these decisions automatic, or are they manually specified by task type? Table 1 includes only two unseen tasks, semantic segmentation and visual grounding, which is not enough to establish automatic adjustment of information granularity. Please explain how the decoding depth is selected, whether target-task tuning is required, and evaluate a broader range of unseen tasks to establish generalization beyond the small set considered in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.