acceptodds
Under review as a conference paper at ICLR 2027

Beyond Multi-modal Alignment: An Information-Preserving Framework for Vision-Language Models

Abstract

Prevalent Vision-Language (VL) alignment techniques in Multi-modal Large Language Models (MLLMs) struggle to adequately align the language model with visual inputs, leading to hallucinations and undermining reliability. We rethink the modality alignment in MLLMs from the perspective of reducing information loss and present an efficient plug-in, VL Superior Alignment (VLSA), that decouples alignment into two stages. The first stage, Perception Alignment, couples compressive high-resolution encoding with reconstructive training through a pretrained text-to-image latent diffusion model (LDM), encouraging compact visual embeddings to preserve input content and align with language. The second stage, Cognition Alignment, introduces auxiliary codebook-index prediction, supplemented by pixel-value prediction, to train the LLM's understanding and use of visual information. Analyses of these two stages yield two empirical insights: reconstruction fidelity alone does not fully characterize the usefulness of compact visual representations for language tasks, and the LLM's utilization of visual information can improve even when the representations are held fixed. Extensive experiments across over 30 MLLM benchmarks and 9 MLLM architectures, thorough ablations, and analyses of computational overhead demonstrate that VLSA consistently improves performance over high-resolution baselines while maintaining inference costs close to those of low-resolution models, with further gains when the computational budget is increased. In service to the MLLM research community, our code and model checkpoints will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Fine-tuning, Self-supervised Learning, Image Understanding, Visual-Question Answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6UAfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces VLSA, a framework motivated by information loss during visual input processing, encoding, and language decoding. Perception Alignment uses low-resolution visual features to query high-resolution crop features through an SA-Perceiver, retaining 576 visual tokens in the default LLaVA configuration, and applies SD3-based reconstruction supervision to the visual encoder and alignment module. Cognition Alignment augments instruction tuning with prediction of VQ-VAE codebook indices and RGB values. The paper reports downstream and efficiency evaluations on LLaVA-based models, additional integrations into other model families, component ablations, and controlled probes of reconstruction recoverability and visual-information utilization.

Critical strengths
  1. The method addresses a practically relevant efficiency–quality trade-off. Retaining high-resolution visual content within a 576-token budget is useful for deployment, and the reconstruction modules are removed at inference. The extensive task evaluations and component ablations provide meaningful evidence of practical benefits. I also appreciate that the cross-architecture experiments distinguish matched baseline comparisons from published reference scores.

  2. The fixed-representation utilization experiment is a substantive diagnostic contribution. In Table 8 and Appendix A.17, both LLMs receive identical visual tensors and the same image shuffles. Cognition Alignment increases the utilization gap by 0.29, with a paired-bootstrap 95% confidence interval of [0.23, 0.35], while accuracy increases from 58.4% to 65.2% on the same 500 questions. This supports improved use of image-specific information under the tested conditions and gives substance to the paper's distinction between representation quality and downstream utilization.

Critical weaknesses
  1. The actual SSFT sequence construction is insufficiently specified to assess the training procedure and its claimed efficiency. Section 3.2 describes predicting RGB values for one quarter of a 336×336 snapshot, corresponding to 84,672 scalar values. Table 40 instead illustrates a quarter of a 672×672 image, corresponding to 338,688 scalar values, alongside a VQ-index sequence. These targets are presented as textual lists, but the paper does not specify their tokenization, actual sequence lengths, context limit, or truncation/chunking and subsampling rules. Equation (10) gives a token-summed loss, but its implementation-level normalization and weighting across auxiliary and QA targets of substantially different lengths remain unclear. Appendix A.13's description of sampling from both template lists for each conversation also needs to be reconciled with the 10:1 task-insertion frequency in Table 27. Without the realized sequence lengths and sampling rules, the training reduction from 35.3 to 18.9 hours in Table 41 cannot be adequately assessed, despite the stated reduction of visual input tokens from approximately 2,880 to 576. Offline target generation does not eliminate the cost of processing the auxiliary tokens during LLM training. This is a reproducibility and cost-accounting concern, rather than evidence that the timing is impossible.

    The placement of auxiliary targets also affects interpretation. Table 40 puts ground-truth VQ indices before a subsequent ordinary QA turn, and A.13 describes random insertion into conversations. Unless attention is explicitly separated or masked, teacher-forced QA training can therefore condition on auxiliary targets that are absent from the usual inference prompt. The paper should clarify the actual masking and quantify whether the benefits persist when this additional target context is excluded.

  2. Important parts of Perception Alignment remain underspecified, limiting verification of its spatial-preservation mechanism and reconstruction supervision. Equation (1) stacks independently encoded crop features in spatial order, but stacking alone does not encode global crop locations: jointly permuting the key and value rows leaves cross-attention unchanged. Low-resolution queries do provide coarse spatial cues; the unresolved issue is how fine details are assigned to global locations when the low-resolution view cannot distinguish them. The paper needs to specify any global crop-position embeddings, coordinates, or correspondence masks used in the implementation.

    The reconstruction interface also requires a more complete description. Equations (3)–(4) and Appendix A.2 do not fully specify the noise/timestep sampling, prediction-target parameterization, and shapes of the prompt and pooled-conditioning branches. Appendix A.17 usefully trains a shared decoder on equally weighted embeddings from both models and evaluates held-out images, but its training-data size/source and full optimization configuration are not given. These omissions prevent independent reproduction of the reconstruction objective and its central diagnostic. They matter because reconstruction supervision is a principal contribution, rather than an incidental implementation choice.

Questions for authors

These questions concern the implemented method and the interpretation of the submitted results; concrete answers could materially improve my assessment.

  1. SSFT implementation: What are the actual tokenized length distributions, maximum context, truncation/subsampling rules, and realized task-insertion frequencies? How is Eq. (10)'s token-summed loss normalized and weighted across auxiliary and ordinary QA targets in the implementation? Is Table 40's RGB target fully used during training? Please reconcile the 336/672 target descriptions and the 10:1 sampling rule, and relate the actual processed tokens and update counts to Table 41's timings.

  2. Auxiliary-context control: Can ordinary QA turns attend to ground-truth VQ/RGB targets during training? If so, do the improvements persist when auxiliary tasks are separate examples, placed after ordinary QA, or isolated by an attention mask? This would help distinguish additional target context from improved utilization of the original visual embeddings.

  3. Spatial-position clarification: How does global crop location enter Equation (1)? Please specify any positional embeddings, coordinates, or correspondence masks. If there are none, what mechanism or evidence supports retention of fine-grained spatial assignments beyond the information already available in the low-resolution queries?

  4. Reconstruction specification: Please provide the SD3 noise/timestep distribution, prediction target, conditioning-branch tensor shapes, and precise meaning of one versus multiple denoising steps during training. What data and optimization schedule were used for the shared decoder in A.17, and how were the two embedding sources mapped into the same conditioning interface?

8 comments

Dear Reviewer 6UAf,

We thank you for the detailed review and for pointing out the implementation details needed to assess the training procedure and its diagnostics.

W1/Q1 — SSFT sequence construction and training cost

We thank the reviewer for identifying the missing implementation details in the SSFT construction. The reviewer is right that the submitted Table 40 and Section 3.2 reflect an early exploratory configuration and are outdated. In the current efficiency-oriented setting, we adopt the same 7× downsampling configuration in predicting RGB values as the early attempt PVP[1], which introduced a similar RGB pixel-value prediction task. Besides, each conversation receives only one of the two auxiliary tasks (The 10:1 sample-level ratio is the best configuration in Table 27 ablations). We have corrected these descriptions in the revised manuscript and apologize for the confusion.

[1] C. Gou, A. Felemban, F. F. Khan, D. Zhu, J. Cai, H. Rezatofighi, and M. Elhoseiny. How Well Can Vision Language Models See Image Details? arXiv:2408.03940, 2024.

Task construction. The final mixture contains 779,289 released conversations (the nominal 790K training set from LLaVA-Next includes about 15K non-released conversations). Each conversation contains exactly one auxiliary task, and the codebook and pixel tasks are never inserted together. The realized counts are 708,445 VQ samples and 70,844 RGB samples, i.e., 10:1 at the sample level. Table 40 previously illustrated an earlier prompt format that included the image size and required output length. Those hints are not used in the current data, and the revised Table 40 now shows the default 336 configuration with the current single-task prompt.

Target sizes. For a 336×336 snapshot, the selected quarter is 168×168. The VQ-VAE produces 21×21 = 441 codebook indices. The RGB target is the same quarter downsampled 7× to 24×24, i.e., 576 pixels and 1,728 scalar values. This follows the 7× downsampling ratio used in PVP[1] but we predict the dense 24×24 region rather than one coordinate at a time.

Token lengths and truncation. The LLaVA-Next experiments use the LLaMA-3 tokenizer. Stage II uses a 32,768-token context. Each VQ answer tokenizes to approximately 1.2k tokens and the 24×24 RGB answer to approximately 2.9k. For the text portion alone (excluding visual and auxiliary-target tokens), the 790K LLaVA-Next training set has a mean length of 173 tokens and a median of 126; the 90th, 95th, and 99th percentiles are 472, 486, and 527 tokens, respectively, and the longest conversation contains 2,693 tokens. Even under the worst-case, the maximum combined sequence is 576 (visual tokens) + 2,693 (the longest text conversation) + 5,184 (the 24×24 RGB answer: 576 pixels × 9 tokens per bracketed [r, g, b] triple, including brackets and separators) = 8,453 tokens; no sample exceeds 32,768, so there are zero truncations and more than 24k tokens of headroom.

Task ratio. The 10:1 ratio is at the sample level. Because an RGB answer is longer than a VQ answer, the realized token-level ratio is 4.09:1: the VQ samples contribute 834.5M answer tokens and the RGB samples contribute 204.0M. The 10:1 ratio therefore controls how often each task appears, not its loss weight.

Loss normalization. Equation (10) defines the token-summed language-modeling loss over the augmented response; we optimize its token-averaged form over all supervised response tokens, so VQ, RGB, and QA tokens receive equal per-token weight and no task-specific coefficient. The 10:1 ratio controls sampling frequency, not loss weight. The reconstruction objective is separate: L_all = L + L_rec (Section 3.2); it is added at the objective level and does not enter the token-level normalization of L.

Training time. With 779,289 conversations, one epoch, and the Stage II batch size of 64 (Table 16), the run has 12,176 updates. The baseline average is 3,053 non-padding tokens (2,880 visual + 173 text), while VLSA averages 2,081 tokens (576 visual + 173 text + about 1,332 auxiliary-target tokens under the 10:1 VQ/RGB mix), corresponding to 2.38B and 1.62B non-padding tokens per epoch. The VLSA average includes these auxiliary tokens, and no auxiliary target is truncated; the reduction is therefore not obtained by dropping them. The 32% shorter sequence reduces the quadratic LLM attention cost by about (2,081/3,053)^2 ≈ 0.46, consistent with the measured Stage II reduction from 30.0h to 14.1h; this measurement includes the single-step LDM/VAE training overhead. Stage I changes only from 5.3h to 4.8h, so the total time falls from 35.3h to 18.9h (Table 41).

Q2 — Auxiliary-context control

The reviewer is right that this distinction matters. In the current construction, each auxiliary pair is inserted into the original conversation at a random position. Under causal attention, ordinary QA turns that follow the inserted pair can attend to the ground-truth VQ/RGB target tokens. Label masking only removes the loss on user tokens; it does not remove those tokens from the attention context. We thank the reviewer for identifying this.

To isolate this effect, we retrained VLSA with each auxiliary task as an independent example, so that ordinary QA and auxiliary targets never occur in the same attention window. The auxiliary objective itself is unchanged; only its co-occurrence with ordinary QA is removed. The same images, targets, and evaluation protocol were used. Due to time constraints, we were able to run this control only once.

Table Q2: Effect of separating auxiliary targets from ordinary QA.

SettingGQASQA-IDocVQA
Full VLSA (auxiliary target in the same conversation)65.377.575.2
Separate auxiliary example65.677.375.3

The differences are within 0.3 points and have no consistent direction. Thus, removing the pathway by which ordinary QA can see ground-truth targets does not materially change performance, and the gains are not primarily explained by target-context leakage. This control complements the fixed-V utilization test (Table 8; Appendix A.17). We have reported both the control and its single-run limitation in the revised manuscript, and have made the insertion rule explicit in Appendix A.13.

W2/Q3 — Global spatial position in the SA-Perceiver

We agree with the reviewer's mathematical point: stacking high-resolution patch features in spatial order does not, by itself, make cross-attention position-aware, because attention is permutation-invariant over the key set. We would like to clarify the intended meaning of the sentence in Section 3.1 — "we gather information from VHiV_{\rm Hi} while preserving their spatial structure according to VLoV_{\rm Lo}." In our design, the low-resolution snapshot is the spatial carrier: its 24×24 queries are position-encoded and provide a coarse spatial anchor. The high-resolution patches are detail carriers: they supply fine-grained content, but they are not intended to carry global crop positions. In the current implementation, the high-resolution keys and values have only CLIP's within-patch position information; the attention mask is a padding/unpad mask, not a correspondence mask. We do not claim fine-grained global assignment of high-resolution patches.

In the case raised by the reviewer, where the low-resolution view cannot distinguish two details, aggregating their high-resolution features into the low-resolution query is harmless (or at least no worse than the naive low-resolution baseline).

To test whether explicit key-side global positions would matter, we trained an ablation variant that adds a parameter-free 2D sincos global position embedding to each high-resolution token. In Table Q3, we observed no material change. Due to time constraints, this ablation was run once.

Table Q3: Ablation with global 2D position embeddings on high-resolution keys.

SettingGQASQA-IDocVQA
Current VLSA65.377.575.2
Ablation: + global key positions65.377.675.0

This indicates that the absence of key-side global positions does not manifest as a material performance degradation on these benchmarks.

In addition, Table 4 (3) vs. (2) isolates the effectiveness of current implementation. Relative to the naive low-resolution baseline (2), row (3) improves ChartQA by 20.5 points (44.8→65.3), DocVQA by 16.4 (45.4→61.8), MME-Perception by 9.2 (1417.3→1426.5), VisWiz by 7.0 (48.9→55.9), and AI2D by 1.9 (67.8→69.7). These are the fine-grained perception tasks for which low-resolution detail is most limiting. The gains show that high-resolution content is effectively injected into the low-resolution query grid; they do not rely on explicit key-side global positions, consistent with the Table Q3 ablation.

Q4 — Reconstruction interface and shared decoder

We have added the following details to Appendix A.2/A.17.

Reconstruction objective. For the input image XX, a frozen SD3 VAE encodes its 336×336336\times336 low-resolution snapshot XLoX_{\rm Lo} into a latent z0∈R16×42×42z_0\in\mathbb R^{16\times42\times42}. Following the rectified flow formulation of SD3 [2], we sample Gaussian noise ϵ∼N(0,I)\epsilon\sim\mathcal N(0,I) and a continuous timestep coordinate u∼LogitNormal(0,1)u\sim\mathrm{LogitNormal}(0,1) (i.e., logit(u)∼N(0,1)\mathrm{logit}(u)\sim\mathcal N(0,1)). We quantize uu to an index k=⌊1000u⌋k=\lfloor 1000u\rfloor and obtain the corresponding tkt_k and σtk\sigma_{t_k} from the SD3 FlowMatchEulerDiscreteScheduler, forming the noisy latent zt=σtkϵ+(1−σtk)z0z_t=\sigma_{t_k}\epsilon+(1-\sigma_{t_k})z_0. The SD3 transformer outputs a velocity vθ(zt,tk,c)v_\theta(z_t,t_k,c); we convert it into the clean-latent estimate zθ(zt,c)=zt−σtkvθ(zt,tk,c)z_\theta(z_t,c)=z_t-\sigma_{t_k}v_\theta(z_t,t_k,c) and supervise it against z0z_0 with the unweighted MSE in Eq. (3), Lrec=∥zθ(zt,c)−z0∥2\mathcal L_{\rm rec}=\|z_\theta(z_t,c)-z_0\|^2. Each training example uses one randomly sampled timestep and one transformer forward pass. In the 2-/5-step ablation in Appendix Table 28, we instead unroll kk denoising updates per iteration and backpropagate through the final loss; this increases training time from 18.9h to 22.1/28.4h with negligible downstream change, so all main results use one step.

[2] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML, 2024.

Conditioning interface. The SA-Perceiver produces V∈R576×4096V\in\mathbb R^{576\times4096} and P∈R4096P\in\mathbb R^{4096}; VV is the visual embedding input to the LLM. Epigone is a low-rank MLP with a SiLU activation, applied to each token of [V;P][V;P]; it has 4096-d input, 256-d hidden, and 4096-d output (in Appendix A.2, Implementation Details, the Epigone sentence reports "4086"; this is a typo for 4096), yielding B×577×4096B\times577\times4096. The first 576 tokens (from VV) form the prompt branch, B×576×4096B\times576\times4096, which is passed directly to SD3's joint attention without padding; the final token (from PP) is projected by a linear head to the B×2048B\times2048 pooled embedding. Together, these form the conditioning input cc in Eq. (4) and match the prompt-plus-pooled interface required by SD3.

Shared decoder (A.17). We use training data from the LLaVA-Next 790K mixture; the 1,000 Q-Bench evaluation images are a separate held-out set and were not used in either VLSA or decoder training. For each training example, we extract one LLaVA-Next embedding (Nbase×4096N_{\rm base}\times4096) and one VLSA embedding (576×4096576\times4096) from the frozen pipelines, pair each embedding with the same target image, and train one SD3 decoder initialized from the official pretrained weights. To map both sources into the same interface, we use a shared token-wise projection Wtok∈R4096×4096W_{\rm tok}\in\mathbb R^{4096\times4096}. For a source embedding Es∈RNs×4096E_s\in\mathbb R^{N_s\times4096} with s∈{base,VLSA}s\in\{\mathrm{base},\mathrm{VLSA}\}, this projection yields the prompt sequence Hs=EsWtok∈RNs×4096H_s=E_sW_{\rm tok}\in\mathbb R^{N_s\times4096}. We average HsH_s over its NsN_s tokens and apply a shared pooled projection Wpool∈R4096×2048W_{\rm pool}\in\mathbb R^{4096\times2048} to obtain the pooled vector ps∈R2048p_s\in\mathbb R^{2048}. Both sources use the same WtokW_{\rm tok} and WpoolW_{\rm pool}; their prompt sequences differ only in length (NbaseN_{\rm base} vs. 576). In SD3's MMDiT blocks, joint attention is applied over the noisy latent tokens ztz_t and the prompt sequence HsH_s, while the pooled vector psp_s is supplied separately as SD3's pooled conditioning. Because joint attention accepts prompt sequences of different lengths, we do not align the two sources to a common token count; each HsH_s is consumed at its native length (with standard batch padding/masking when needed), while psp_s always has the same 2048-d shape. All reconstruction-side parameters, including both projections and the denoiser, are shared. The two pairs are weighted equally, L=0.5(Lbase+LVLSA)\mathcal L=0.5(\mathcal L_{\rm base}+\mathcal L_{\rm VLSA}), and are optimized with the same flow-matching objective [2] using one denoising step per iteration. Training uses AdamW with a learning rate of 10−310^{-3}, a batch size of 128 examples (256 pairs), a cosine schedule with 3% warmup, zero weight decay, and bf16 precision for 3 epochs without early stopping. The VAE remains frozen, while the denoising transformer and conditioning projections are trainable. At evaluation, the trained decoder is held fixed and applied to both embeddings of these images.

Closing

We appreciate the reviewer’s emphasis on implementation-level clarity. The revised manuscript separates what is measured by the reconstruction objective, what is measured by the fixed-representation utilization test, and what remains an implementation choice.

Reviewer_M8weOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VLSA, a framework to improve vision-language alignment by reducing information loss during visual encoding and reasoning. Experiments across more than 30 benchmarks and 9 MLLM architectures show consistent improvements while reducing computational costs. The experimental evaluation is extensive and convincing, although the methodological novelty is moderate, as the framework mainly combines existing techniques.

Critical strengths
  1. Extensive experimental validation. The framework is evaluated across more than 30 benchmarks and 9 MLLM architectures, showing consistent improvements in visual understanding and reasoning. This broad evaluation provides strong evidence of the effectiveness and generalizability of the proposed approach.

  2. Effective visual information preservation with reduced computational cost. VLSA successfully combines compact visual representations with auxiliary reconstruction and prediction objectives, achieving competitive or improved downstream performance while significantly reducing the number of visual tokens. The ablation studies further support the benefits of the proposed components.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines existing techniques and auxiliary prediction objectives. While their integration is effective, the individual components are not fundamentally novel, and the paper provides limited new theoretical insights into multimodal alignment.

  2. Limited justification of the information-loss framework and individual contributions. The distinction between Original, Encoding, and Decoding Information Loss is conceptually interesting but lacks a rigorous theoretical formulation. Furthermore, although the ablation studies demonstrate improvements, they do not fully isolate the contribution of the pretrained diffusion model from that of the reconstruction objective itself. This makes it difficult to determine which aspects of the framework are primarily responsible for the reported gains.

Questions for authors
  1. How much of the reported improvement can be attributed specifically to the pretrained diffusion prior, rather than to the auxiliary reconstruction objective itself? Could the authors provide a comparison with a simpler reconstruction-based approach under comparable training conditions? This would help clarify whether the additional complexity of the diffusion-based component is justified.

  2. The distinction between Encoding Information Loss and Decoding Information Loss is conceptually interesting but not rigorously established. Can the authors provide stronger empirical evidence that these represent distinct sources of performance degradation, rather than different manifestations of the same underlying limitations? This would strengthen the central motivation of the framework.

  3. The ablation studies show that some components improve certain benchmarks while degrading others. Can the authors give their own interpretation about this?

5 comments

Dear Reviewer M8we,

We thank you for the careful review and constructive suggestions.

W1 — Limited methodological novelty

We agree that the ingredients of VLSA—multi-resolution encoding, visual reconstruction, and auxiliary visual supervision—have been previously studied. However, most prior methods target only one stage of information loss. Their implementations also have specific limitations beyond incomplete stage coverage: Q-former-style resampling can worsen E-IL; Ross places reconstruction on LLM-processed tokens, which can interfere with LLM training; and grounding/segmentation-based auxiliary objectives cover only predefined visual attributes and require manual or semi-automatic annotation (Section 2; Appendices A.4.3 and A.12; Table 23 in Appendix A.3). These gaps concern how the ingredients are instantiated and coordinated, not whether they exist.

Against this background, the first level of our novelty is the formulation itself. We define information loss along X → (X_Hi, X_Lo) → V → response and distinguish O-IL (input), E-IL (encoding), and D-IL (decoding) as stage-localized and testable. This formulation broadens modality alignment from embedding matching to stage-wise information preservation. It is not a relabeling: it determines where to intervene and motivates Perception Alignment and Cognition Alignment. VLSA coordinates these interventions across stages, and Table 4 shows their complementary effects.

Against this formulation, the second level of novelty is technical: the two core implementations introduce new training interfaces. First, our reconstructive training is novel in where and what the pretrained text-to-image prior supervises (Section 3.1): it regularizes the final pre-LLM visual embeddings for both reconstructable input content and language-compatible semantics, with gradients updating both visual encoder and projector. Second, our auxiliary objective is novel in introducing a new self-supervised training task: we require the LLM to predict textualized VQ-VAE codebook indices of local image regions and RGB values of a sampled region, using the standard language-modeling loss. Unlike annotation-based grounding/segmentation objectives, which cover only predefined attributes, this task is generated automatically from the image, covers discrete high-level semantics and low-level appearance, and requires no manual labels, vocabulary expansion, architecture change, or inference overhead. To our knowledge, this specific task has not been proposed in understanding-oriented MLLM SFT.

Beyond the formulation and implementation, the paper contributes empirical insights that are not implied by the individual components. First, multimodal alignment has a utilization gap, not merely a representation gap: a representation can contain the relevant information yet be underused by the LLM, and this underuse is trainable (complementary analyses in Appendix A.16, Table 8 and Appendix A.17 support it). Second, reconstruction quality alone is a poor predictor of downstream usefulness (Tables 19 and 29; Appendix A.19). Third, auxiliary visual supervision is complementary rather than interchangeable: pixel-value targets alone emphasize low-level details and can degrade high-level semantic understanding. The joint objective is best only when they are paired with codebook-index targets (Table 7).

In the revised manuscript, we have made the two-level limitation/novelty structure explicit in the Introduction, highlighted these insights in the Discussion, and added a compact task-definition table contrasting the new objective with grounding/segmentation objectives in Appendix A.4.

W2 — Limited justification of the information-loss framework and individual contributions

Framework justification. The paper does not claim a formal axiomatic theory of information loss; it proposes an operational framework with stage-specific, testable predictions. Along X → (X_Hi, X_Lo) → V → response, O-IL, E-IL, and D-IL localize information loss at the input, encoding, and response-generation stages. "Separable" means that these stages can be tested with stage-appropriate diagnostics, not that the losses are statistically independent, additive, or addressed one-to-one by individual components. O-IL is diagnosed by the input-resolution/PSNR/GQA analysis in Table 31; E-IL is diagnosed by the shared-decoder reconstruction test (12.2 vs. 21.8 dB; Table 8/Appendix A.17) and the frequency-domain analysis (Table 12); and D-IL is diagnosed by the fixed-representation utilization test (ΔU=0.29; 58.4%→65.2%; Table 8/Appendix A.17), together with the attention/CKA diagnostics in Tables 42 and 43. Compressive encoding, reconstructive training, and Cognition Alignment primarily target O-IL, E-IL, and D-IL, respectively, while interacting; Table 4 shows their complementary effects. The revised Section 3 and Appendix A.4 will state this scope explicitly.

Pretrained prior vs. reconstruction objective. We separate these factors with three comparisons. Table 4 (3)→(4) holds the compressed representation and token budget fixed and adds reconstruction, isolating the reconstruction objective. Tables 45 and 46 compare pretrained versus randomly initialized SD3 with the reconstruction objective and architecture fixed, isolating the effect of the pretrained diffusion prior. Tables 19 and 29 compare LDM-based reconstruction with MAE/VAE/VQ-VAE alternatives, evaluating the reconstruction model. The pretrained-vs-scratch gap grows from +0.3/+1.2/+0.5 at 336×336 to +3.5/+3.8/+2.0 at 672×672 (Table 45), and pretrained SD3 yields higher vision-language CKA in five of six layers (Table 46, Layer 12 +0.21).

Attribution of the reported gains. These comparisons support a two-factor attribution: the reconstruction objective supplies the E-IL-reducing training signal even with scratch SD3, whereas pretrained SD3 additionally contributes its learned text-to-image conditioning prior, an advantage that becomes more visible when O-IL is reduced. The reported gains are therefore primarily due to their combination, not to either factor alone. We do not claim that the prior alone explains the full gain, nor that the two factors are independent.

Q1 — Attribution to the pretrained diffusion prior versus the reconstruction objective

We thank the reviewer for this question. Table 45 (Appendix A.19) already reports a direct comparison between scratch SD3 and pretrained SD3 for VLSA performance, with the reconstruction objective and architecture fixed. The pretrained prior adds a modest gain at 336×336 (+0.3/+1.2/+0.5 on GQA/SQA-I/DocVQA) and a larger gain at 672×672 (+3.5/+3.8/+2.0). Scratch SD3 already performs competitively under the reconstruction objective, so the prior adds a further gain rather than explaining the full improvement. This attribution is detailed in W2, which separates the prior from the reconstruction objective using the ablation in Section 4.2 (Table 4), the LDM-alternative comparison in Table 19 (Appendix A.3), and the pretrained-versus-scratch comparisons in Tables 45/46.

The reviewer also asks for a comparison with a simpler reconstruction approach under comparable training conditions. Table 19 (Appendix A.3) and Section 4.2 already report this comparison: under the same VLSA training pipeline, with only the reconstruction module replaced, the pretrained LDM is strongest overall among the tested VAE, VQ-VAE, MAE, and scratch-SD3 variants. It is stronger on GQA/SQA-I (65.3/77.5 vs. 64.5/74.3 for MAE), while MAE is slightly better on DocVQA (75.4 vs. 75.2). Despite slightly lower PSNR (20.18 vs. 20.76), the LDM yields higher vision-language CKA than MAE in five of six layers (Table 29, Appendix A.3). Thus, a simple reconstruction objective alone does not reproduce the gain. The pretrained text-to-image prior contributes language-compatible semantics.

Finally, Appendix A.4 and Section 4.1 show that the additional complexity is confined to training: the LDM uses a single denoising step per iteration and is removed at inference, adding no inference overhead. The shorter visual sequence more than offsets its training cost (18.9h vs. 35.3h for LLaVA-Next).

Q2 — Are Encoding Information Loss and Decoding Information Loss distinct?

The distinction is operational and testable: E-IL is information not retained in the visual representation V, whereas D-IL is information retained in V but not used by the LLM. We test these two stages separately.

For E-IL, Section 4.1 and Appendix A.17 report a shared-decoder reconstruction test (Table 8). We freeze the LLaVA-Next and VLSA visual pipelines, extract their visual representations for every image used to train the decoder, and train one SD3 decoder, initialized from the official pretrained weights, on the two equally weighted representation–image pairs. All reconstruction-side parameters, including the conditioning projections, are shared. We then hold the decoder fixed and reconstruct the same 1,000 Q-Bench images, none of which appeared in VLSA or decoder training. PSNR is 12.2 dB for LLaVA-Next and 21.8 dB for VLSA. Because the decoder, evaluation images, and pretrained prior are identical, this gap isolates the encoding stage: it reflects information that VLSA's encoder/projector retains in V but LLaVA-Next's does not, which is the operational definition of E-IL. It cannot be attributed to decoder capacity, training-image overlap, or the LLM's utilization.

For D-IL, Section 4.1 and Appendix A.17 report a correct-versus-shuffled utilization test (Table 8). We randomly sample 500 Q-Bench questions and use the checkpoint trained without Cognition Alignment to extract each question's visual representation V_i. We feed that same V_i to two LLMs—one trained with Cognition Alignment and one without—so both receive identical visual information. We then pair each question with three shuffled representations from other images, using the same permutations for both LLMs. The matched-image score is the correct-option log-odds with V_i. The shuffled-image score averages the same quantity over the three mismatched representations. U is their difference, and ΔU is the mean paired difference in U between the two LLMs. Cognition Alignment raises the matched-image score by 0.30 but the shuffled-image score by only 0.01, yielding ΔU=0.29 [0.23,0.35]. Matched-image accuracy rises from 58.4% to 65.2%. Because V and the shuffles are identical, this gap isolates the decoding stage: it reflects how the LLM uses information already present in V, which is the operational definition of D-IL. It cannot be attributed to differences in V content or encoding quality.

Together, these tests form an operational dissociation: the E-IL test varies V with the reconstruction decoder fixed, while the D-IL test varies the LLM's use of V with V fixed. Each difference cannot be explained by the other stage, supporting E-IL and D-IL as distinct sources of degradation in the tested settings.

Q3 — Mixed ablation results

This is an interesting and meaningful question. We discussed this mixed pattern in Section 4.2, where Table 4 shows that each component's effect depends on which stage remains the limiting factor. We interpret a sign change in an individual metric as expected rather than evidence of instability; the components are complementary interventions. For convenience, we summarize the main points below.

Compressive encoding aggregates high-resolution patches into a fixed 576-token low-resolution grid, reducing O-IL at a controlled token budget. It can, however, introduce E-IL through compression. Table 4 reports the relevant ablation. Relative to the high-resolution baseline, the compressive-encoding-only variant lowers ChartQA/DocVQA/MME-perception by 1.8/11.9/29.2 points (Table 4, (3)). Adding reconstructive training, which requires the compressed embeddings to reconstruct the input through a pretrained LDM, recovers these losses to +0.3/+1.8/+25.7 (Table 4, (3)→(4)). The two components therefore address different stages and are complementary, not interchangeable.

Cognition Alignment targets a different stage. It augments standard SFT with codebook-index and pixel-value prediction, training the LLM to use visual information already present in V. Table 4 reports both its isolated and combined variants. Used alone, it improves six of seven metrics but retains the Any-resolution cost and does not address O-IL/E-IL in visual encoding (Table 4, (5)). Combined with full VLSA, it improves most metrics under the 336* setting by helping the LLM use the information that Perception Alignment has retained (Table 4, (4)→(6)). Its benefit is therefore complementary to, rather than a substitute for, the earlier stages. At the objective level, Table 7 compares no SSFT, codebook-only, pixel-only, and joint objectives: pixel-only supervision can hurt SQA, consistent with a dense RGB target competing with language modeling, whereas codebook-only and the joint objective are consistently better. Each decrease is therefore consistent with a stage or objective that the corresponding variant leaves unaddressed, rather than a failure of the full design.

Reviewer_HnGHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies visual information loss during encoding and the LLM's use of visual representations. VLSA combines high-resolution feature aggregation and diffusion-based reconstruction supervision in Perception Alignment, then adds codebook-index and pixel-value prediction during instruction tuning in Cognition Alignment. The default LLaVA configuration sends 576 visual tokens to the LLM. The contribution is the combination of these objectives with compact visual representations, supported by experiments across several architectures and controlled analyses of recoverability and utilization.

Critical strengths
  1. The fixed-representation experiment in Appendix A.17 is a useful analysis. Both LLMs receive identical visual tensors and the same shuffled-image controls. Cognition Alignment increases the utilization gap by 0.29, with a question-level paired-bootstrap 95% confidence interval of [0.23, 0.35]; accuracy on the same 500 questions rises from 58.4% to 65.2%. This gives more direct evidence of improved visual-information use than attention maps alone.
  2. The default configuration offers a useful accuracy-cost tradeoff. For LLaVA-Next, GQA, SQA-I, and DocVQA improve from 64.9/74.6/73.7 to 65.3/77.5/75.2 with a shorter visual sequence. The reconstruction modules are removed at inference, and Table 41 reports a reduction in total training time from 35.3 to 18.9 hours under the authors' setup.
  3. The evaluation includes several relevant controls: alternative projectors, MAE reconstruction, a ROSS-style reconstruction location, and matched fine-tuning for Qwen2.5-VL and InternVL-2.5. Appendix A.18 distinguishes matched baselines from official results. The shared-decoder experiment also strengthens the reconstruction evidence by holding the decoder and unseen evaluation images fixed. The paper appropriately notes that these are operational tests under specific protocols, and that reconstruction fidelity alone does not determine downstream usefulness.
Critical weaknesses
  1. The component comparisons need clearer experimental conditions. The paper already acknowledges the relationship to Mini-Gemini and includes compressed projector alternatives. For Table 18, please list each projector's output-token count and confirm whether reconstruction and Cognition Alignment objectives are held fixed. The Q-former description refers to the same dimensionality as the low-resolution representations; stating the actual query count would make the budget comparison unambiguous. Table 37 similarly needs an explicit statement of the denoiser initialization and objectives used for its ROSS-style variant. The surrounding discussion contrasts VLSA's pretrained denoiser with original ROSS's random initialization, but does not clearly distinguish that comparison from the implementation of the table's variant. These details would help establish which gains come from the proposed architecture and which come from its supervision. Clarifying the existing experiments should take priority over adding more baselines. For the method's positioning, DIVA [1] is relevant to learning visual representations from diffusion feedback and deserves discussion. Its CLIP post-training setting differs from VLSA's joint VLM training. A clear account of this relationship would be sufficient here; a full reproduction is not necessary for the current comparison.

[1] Wenxuan Wang et al. Diffusion Feedback Helps CLIP See Better. arXiv:2407.20171, 2024. https://arxiv.org/abs/2407.20171

  1. Hyperparameter selection and uncertainty need clarification. Table 27 selects the 10:1 codebook-to-pixel ratio based on AI2D, SQA and ChartQA results, but does not identify the split used for selection. Please state how this ratio and the loss weights were chosen, and whether the selection data were separate from the final evaluation data. The SQA range of 67.9 to 78.6 shows sensitivity to the task mixture and makes the selection protocol worth documenting. Table 26 reports three VLSA seeds and single-run baseline references, and explicitly acknowledges that this assesses VLSA's stability rather than the significance of paired gains. Paired baseline/VLSA runs on a few principal benchmarks would strengthen the evidence for smaller improvements without requiring the full evaluation suite to be repeated.

  2. Component comparisons need clearer conditions Table 18 should list output-token counts, Q-former query counts, and whether reconstruction and CA supervision are shared. Table 37 should specify the ROSS-style variant's denoiser initialization and objectives; the discussion of the original ROSS does not establish this internal variant's configuration. These details affect component attribution. Position PA against DS-VLM's connector supervision (Sun et al.) and DIVA's CLIP-only diffusion feedback (Wang et al.). DS-VLM conditions on encoder/connector representations before the LLM; DIVA freezes diffusion and updates CLIP from image-only inputs. Specify the increment from compressed conditioning, SD3, and CA. The remaining contribution is the particular compression/CA combination, rather than diffusion supervision itself.

  3. Table 20 lists DocVQA as 75.2 for both CLIP rows while claiming a gain of 1.5; Table 4 gives the baseline as 73.7. Please reconcile these entries. Also clarify the configuration of Table 31: its input-resolution experiment gives GQA 65.2 at 672 by 672, whereas the snapshot-resolution experiments in Tables 5, 24, and 25 give 69.2. These may be different settings and should be identified as such.

  4. Figure 3 removes text instructions and measures image-only forward passes. Appendix A.5 acknowledges other runtime factors, but deriving a 25-fold latency reduction from quadratic attention scaling still overstates what that calculation establishes. Please distinguish attention computation from total runtime and limit the efficiency claim to the measured setting. A generation-time measurement with specified prompt and output lengths would be a useful extension.

Questions for authors

The rebuttal should prioritize clarifying token budgets and supervision settings in Table 18, denoiser initialization and objectives in Table 37, and the data splits used for hyperparameter selection. The authors should also reconcile the numerical and configuration inconsistencies and restrict efficiency claims to what the measurements establish. Paired baseline/VLSA runs on a few principal benchmarks would strengthen the evidence for smaller improvements, but clarifying the existing comparisons is the higher priority. Convincing responses could move my recommendation toward acceptance.

4 comments

Dear Reviewer HnGH,

We thank you for the detailed review and for identifying the specific conditions needed to make the comparisons verifiable.

W1 — Component comparisons need clearer experimental conditions

We thank the reviewer for identifying the missing conditions in the projector and ROSS-style comparisons. The revised captions will report them explicitly.

Table 18. The five rows differ only in the visual projector; reconstruction and Cognition Alignment are held fixed. Q-Former and SA-Perceiver both output a fixed 576-token sequence. The other three projectors operate on the LLaVA-Next's any-resolution visual sequence: let N(X)N(X) denote its token count for image XX, which depends on the selected patch grid. MLP and Self-Attn preserve the full sequence N(X)N(X); Pixel-Unshuffle reduces it to N(X)/4N(X)/4. Thus, MLP and Self-Attn are full-token references, Pixel-Unshuffle is a quarter-token compression reference, and the only strict budget-matched comparison is Q-Former (576 queries) versus SA-Perceiver (576 tokens). Across our evaluation set, the LLaVA-Next visual sequence averages about 2,880 tokens, but N(X)N(X) varies with image size and aspect ratio.

Table 37. The ROSS-style variant uses the LLM outputs as reconstruction cues, is initialized from the same pretrained SD3 denoiser, and uses the same reconstruction objective and Cognition Alignment as VLSA; only the cue position changes (LLM outputs versus projector outputs). The comparison therefore isolates the reconstruction cue rather than denoiser initialization or supervision. The surrounding discussion of the original ROSS's random initialization is not used as the configuration of this internal variant.

DIVA. We thank the reviewer for pointing us to DIVA (Wang et al., 2024). DIVA uses diffusion feedback to improve the CLIP encoder's detail retention in an image-only post-training setting, with the diffusion model frozen. VLSA shares the high-level idea of using reconstruction as a training signal, but differs in the training interface: we use the final pre-LLM visual embeddings as the reconstruction cue. Consequently, the reconstruction loss updates both the vision encoder and the visual adapter, so it targets adapter-induced E-IL as well as encoder-side detail retention, rather than only the CLIP encoder. Combined with joint training of the VLM and denoising transformer, this interface allows our model to exploit the pretrained text-to-image LDM's vision-language alignment (Appendix A.4.2 and A.19 (Tables 29 and 46) experimentally confirm this effect).

W2 — Hyperparameter selection and paired baseline comparison

We clarify two separate choices below: the 10:1 codebook-to-pixel ratio, which controls the relative frequency of the two auxiliary tasks, and the 1:1 loss weight, which balances reconstruction and language modeling. We then report paired baseline/VLSA runs on three benchmarks where the gains are small, to assess whether these gains persist across seeds.

Codebook-to-pixel ratio. The 10:1 ratio is a sample-level insertion frequency, not a loss weight: our training mixture contains ten codebook-index conversations for every one pixel-value conversation. Table 27 reports all five variants (1:2, 1:1, 2:1, 5:1, and 10:1), each trained on the same LLaVA-Next 790K training set and evaluated on AI2D, SQA, and ChartQA. We did not use a separate validation split, so the selection and evaluation data overlap. This overlap is a limitation of the protocol, but it does not materially affect the qualitative conclusion: across all three benchmarks, 5:1 and 10:1 are within 1.4 points of each other, and both outperform the lower ratios. We therefore treat the sweep as a design ablation rather than a held-out selection.

The 10:1 ratio is a single global setting, not a value tuned per benchmark: it is used for every benchmark and architecture, and all five variants are reported. Performance improves steadily on AI2D and SQA as codebook prediction becomes more frequent, with 10:1 best on both. ChartQA peaks at 5:1. The SQA range of 67.9–78.6 is the spread across these five variants, not across seeds. We adopt 10:1 as a practical balance, not a uniquely optimal value.

Loss weight. The 1:1 weight is the ratio between reconstruction and language-modeling losses. The auxiliary targets share the same token-level language-modeling loss and introduce no task-specific weights. The loss-weight sweep follows the same protocol: Table 22 reports four settings (0.25:0.75, 0.5:0.5, 0.75:0.25, and 1:1) trained on the 790K mixture and compared on AI2D, ChartQA, and DocVQA, with no separate validation split. The 1:1 setting is best on all three benchmarks (71.4/67.9/75.2), and the closest alternative is within 0.4 points on each benchmark. As with the ratio, one setting is used for all benchmarks and architectures. The top choice is consistent across all three, so the selection is not driven by a single metric. We present it as an empirical default, not a uniquely optimal value.

Uncertainty. We re-ran the LLaVA-Next AnyRes baseline with the same three seeds as VLSA (42/456/2024) under the same data and evaluation pipeline. Only the seed varies across these three baseline runs. Table W2 reports per-seed values and paired differences on GQA, VQA-v2, and POPE.

Mean paired gains are +0.50/+1.14/+1.23. The baseline standard deviations across the three seeds are 0.25/0.21/0.15, comparable to VLSA's 0.21/0.15/0.40. The revised Table 26 will report baseline and VLSA means and standard deviations from these paired runs. The appendix will list per-seed values and state the selection protocol, including the absence of a separate validation split.

Table W2: Paired LLaVA-Next AnyRes baseline and VLSA results across three seeds.

MetricBaseline (42/456/2024)VLSA (42/456/2024)Δ (42/456/2024)Mean Δ ± std95% CI
GQA64.9 / 64.6 / 65.165.30 / 65.20 / 65.61+0.40 / +0.60 / +0.51+0.50 ± 0.10[0.25, 0.76]
VQA-v282.4 / 82.5 / 82.183.50 / 83.31 / 83.60+1.10 / +0.81 / +1.50+1.14 ± 0.35[0.27, 2.00]
POPE87.7 / 87.4 / 87.588.60 / 88.46 / 89.22+0.90 / +1.06 / +1.72+1.23 ± 0.43[0.16, 2.30]

Δ = VLSA − baseline. The intervals are two-sided 95% t-intervals (df = 2). With only three seeds, we treat them as descriptive summaries, not as a well-powered significance test.

W3 — Component attribution and positioning against DS-VLM/DIVA

We thank the reviewer for bringing DS-VLM (Sun et al., ICML 2025) to our attention. It is a highly relevant reference for our pre-LLM reconstruction objective, and we appreciate the opportunity to clarify the relationship. The revised manuscript will discuss it alongside DIVA. W3 also overlaps with W1 on the Table 18/Table 37 conditions and DIVA. We refer to our W1 response for the full details and summarize the key controls below.

DIVA and VLSA. Please refer to our W1 response.

DS-VLM and VLSA. DS-VLM provides a fine-grained scheme for conditioning a diffusion model on visual-encoder/connector outputs, creating a short-path gradient from pixel space to visual features while retaining standard text supervision. This shows that diffusion reconstruction can directly improve visual feature quality. Together with DIVA, it establishes the value of diffusion supervision. We agree that diffusion supervision or the pretrained LDM itself is not our contribution. Our differences lie in what the reconstruction loss is used for and where it is applied.

DS-VLM primarily uses the reconstruction loss to improve the quality of the encoder representation. VLSA uses the reconstruction loss for two additional purposes. The first is to improve the integration and retention of high-resolution features from the original image, in conjunction with compressive encoding (supported by Table 18, Table 4 (3) vs (2), and Table 4 (4) vs (3)). The second is to leverage the text-to-image LDM prior, whose text-image pretraining encourages the final pre-LLM embedding to preserve image content in a form compatible with language (supported by Table 19, Table 29, and Tables 45/46).

Our reconstruction training therefore contributes in two ways: it improves high-resolution feature integration and retention under compressive encoding, and it uses the text-to-image LDM prior to improve the vision-text semantic alignment of pre-LLM features. Cognition Alignment adds codebook-index and pixel-value prediction as an LLM-side objective, improving the LLM's use of retained visual information (Table 8/A.17). The full contribution is therefore this Perception Alignment combination plus CA, not diffusion supervision itself.

Comparison controls. As detailed in our W1 response, Table 18 holds reconstruction and CA fixed across projector variants. Q-Former and SA-Perceiver both use 576 tokens, so they form the only strict budget-matched comparison. MLP/Self-Attn are full-token references, and Pixel-Unshuffle is a quarter-token reference. Table 37 changes only the reconstruction cue (LLM outputs vs. projector outputs) while keeping the pretrained SD3 initialization and objective fixed. These controls isolate the architecture and supervision effects the reviewer asks about.

W4 — Numerical and configuration inconsistencies

The reviewer is correct about Table 20. The CLIP baseline DocVQA entry of 75.2 was a transcription error; the correct value is 73.7. The VLSA entry (75.2) and the +1.5 gain are unchanged and consistent with Table 4; the other entries and deltas in the table are unaffected. The revised Table 20 will use 73.7 for this baseline cell.

Table 31 reports a different setting. It normalizes the original input image to 168×168, 336×336, or 672×672 under a fixed model and training recipe, and measures reconstruction PSNR and GQA as an O-IL diagnostic. It is not the VLSA snapshot-scaling experiment: in Tables 5, 24, and 25, the Res. column denotes the low-resolution snapshot X_Lo and the associated visual-token budget, and the 672×672 row there is the full VLSA setting that yields GQA 69.2. The 672×672 row in Table 31 is an input-normalized diagnostic point and yields GQA 65.2. These two rows therefore measure different interventions and should not be compared directly. The revised caption will state this distinction explicitly.

W5 — Efficiency claims

Appendix A.5's 25× figure was intended as a theoretical upper bound on the prefilling latency term—specifically, the attention computation over the input sequence—not as a total generation-time estimate. Under the simplifying assumptions used in A.5, visual tokens are the only changing term and attention dominates the LLM computation, so the prefill attention cost can be reduced by at most 25×. It is not a measured end-to-end or generation-time speedup. We agree that the original wording ("overall system latency") did not make this scope explicit and could be misread; the revised manuscript will relabel it as a prefill-time upper bound and state its assumptions.

To provide a practical generation-time reference, we measured total generation time on 50 randomly sampled LLaVA-Next 790K questions. The average prompt length was 198 tokens. Averaged over the 50 questions, LLaVA-Next generated 93 tokens and took 2.99s per question, while VLSA generated 101 tokens and took 1.84s per question, on one A100 80GB with batch size 1 (single run).

Table W5: Average total generation time on 50 LLaVA-Next 790K questions (single run).

ModelAvg. prompt lengthAvg. output lengthAvg. total generation time
LLaVA-Next198932.99 s
VLSA (336*)1981011.84 s

This comparison includes both prefill and decoding, and the two models generate different output lengths. The lower average total generation time is therefore a practical generation-time observation, not evidence that the prefill stage alone is faster.

To isolate the visual-path cost, Figure 3 reports image-only forward-pass latency and FLOPs for 1,000 images on a single A100 with batch size 1, excluding text instructions, text prefill, and generation; VLSA keeps its 336* snapshot and 576-token visual budget. Under this measured setting, VLSA's latency and FLOPs are comparable to low-resolution processing and lower than the any-resolution baseline. The revised Figure 3 caption will state these conditions explicitly. The efficiency claim is therefore limited to this image-only setting and the visual-token/prefill budget (576 vs. about 2,880 visual tokens).

Questions For Authors

We have clarified the token budgets and supervision settings in Table 18, the denoiser initialization and objectives in Table 37, and the selection protocol for the codebook-to-pixel ratio. We have also added paired baseline runs on GQA, VQA-v2, and POPE, identified the corrections for Table 20/Table 31, and limited the efficiency claim to the measured image-only setting.

We hope these clarifications address the reviewer’s concerns.

Dear Reviewer HnGH,

Thank you for your earlier detailed comments. We have sought to address your questions in the rebuttal, and the revised PDF reflects these changes:

  • Table 18/37 now state token budgets and supervision settings;
  • The ratio and loss-weight selection protocol is documented, including that there is no separate validation split;
  • Table 26 adds paired baseline/VLSA runs on GQA, VQA-v2, and POPE across three seeds;
  • Table 20 is corrected to 73.7, and Table 31 is distinguished from snapshot scaling;
  • The efficiency claim is limited to prefilling latency, with assumptions stated, and a generation-time reference is added;
  • DS-VLM and DIVA are explicitly positioned.

We would appreciate your feedback when convenient.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.