- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces VLSA, a framework motivated by information loss during visual input processing, encoding, and language decoding. Perception Alignment uses low-resolution visual features to query high-resolution crop features through an SA-Perceiver, retaining 576 visual tokens in the default LLaVA configuration, and applies SD3-based reconstruction supervision to the visual encoder and alignment module. Cognition Alignment augments instruction tuning with prediction of VQ-VAE codebook indices and RGB values. The paper reports downstream and efficiency evaluations on LLaVA-based models, additional integrations into other model families, component ablations, and controlled probes of reconstruction recoverability and visual-information utilization.
Critical strengths
-
The method addresses a practically relevant efficiency–quality trade-off. Retaining high-resolution visual content within a 576-token budget is useful for deployment, and the reconstruction modules are removed at inference. The extensive task evaluations and component ablations provide meaningful evidence of practical benefits. I also appreciate that the cross-architecture experiments distinguish matched baseline comparisons from published reference scores.
-
The fixed-representation utilization experiment is a substantive diagnostic contribution. In Table 8 and Appendix A.17, both LLMs receive identical visual tensors and the same image shuffles. Cognition Alignment increases the utilization gap by 0.29, with a paired-bootstrap 95% confidence interval of [0.23, 0.35], while accuracy increases from 58.4% to 65.2% on the same 500 questions. This supports improved use of image-specific information under the tested conditions and gives substance to the paper's distinction between representation quality and downstream utilization.
Critical weaknesses
-
The actual SSFT sequence construction is insufficiently specified to assess the training procedure and its claimed efficiency. Section 3.2 describes predicting RGB values for one quarter of a 336×336 snapshot, corresponding to 84,672 scalar values. Table 40 instead illustrates a quarter of a 672×672 image, corresponding to 338,688 scalar values, alongside a VQ-index sequence. These targets are presented as textual lists, but the paper does not specify their tokenization, actual sequence lengths, context limit, or truncation/chunking and subsampling rules. Equation (10) gives a token-summed loss, but its implementation-level normalization and weighting across auxiliary and QA targets of substantially different lengths remain unclear. Appendix A.13's description of sampling from both template lists for each conversation also needs to be reconciled with the 10:1 task-insertion frequency in Table 27. Without the realized sequence lengths and sampling rules, the training reduction from 35.3 to 18.9 hours in Table 41 cannot be adequately assessed, despite the stated reduction of visual input tokens from approximately 2,880 to 576. Offline target generation does not eliminate the cost of processing the auxiliary tokens during LLM training. This is a reproducibility and cost-accounting concern, rather than evidence that the timing is impossible.
The placement of auxiliary targets also affects interpretation. Table 40 puts ground-truth VQ indices before a subsequent ordinary QA turn, and A.13 describes random insertion into conversations. Unless attention is explicitly separated or masked, teacher-forced QA training can therefore condition on auxiliary targets that are absent from the usual inference prompt. The paper should clarify the actual masking and quantify whether the benefits persist when this additional target context is excluded.
-
Important parts of Perception Alignment remain underspecified, limiting verification of its spatial-preservation mechanism and reconstruction supervision. Equation (1) stacks independently encoded crop features in spatial order, but stacking alone does not encode global crop locations: jointly permuting the key and value rows leaves cross-attention unchanged. Low-resolution queries do provide coarse spatial cues; the unresolved issue is how fine details are assigned to global locations when the low-resolution view cannot distinguish them. The paper needs to specify any global crop-position embeddings, coordinates, or correspondence masks used in the implementation.
The reconstruction interface also requires a more complete description. Equations (3)–(4) and Appendix A.2 do not fully specify the noise/timestep sampling, prediction-target parameterization, and shapes of the prompt and pooled-conditioning branches. Appendix A.17 usefully trains a shared decoder on equally weighted embeddings from both models and evaluates held-out images, but its training-data size/source and full optimization configuration are not given. These omissions prevent independent reproduction of the reconstruction objective and its central diagnostic. They matter because reconstruction supervision is a principal contribution, rather than an incidental implementation choice.
Questions for authors
These questions concern the implemented method and the interpretation of the submitted results; concrete answers could materially improve my assessment.
-
SSFT implementation: What are the actual tokenized length distributions, maximum context, truncation/subsampling rules, and realized task-insertion frequencies? How is Eq. (10)'s token-summed loss normalized and weighted across auxiliary and ordinary QA targets in the implementation? Is Table 40's RGB target fully used during training? Please reconcile the 336/672 target descriptions and the 10:1 sampling rule, and relate the actual processed tokens and update counts to Table 41's timings.
-
Auxiliary-context control: Can ordinary QA turns attend to ground-truth VQ/RGB targets during training? If so, do the improvements persist when auxiliary tasks are separate examples, placed after ordinary QA, or isolated by an attention mask? This would help distinguish additional target context from improved utilization of the original visual embeddings.
-
Spatial-position clarification: How does global crop location enter Equation (1)? Please specify any positional embeddings, coordinates, or correspondence masks. If there are none, what mechanism or evidence supports retention of fine-grained spatial assignments beyond the information already available in the low-resolution queries?
-
Reconstruction specification: Please provide the SD3 noise/timestep distribution, prediction target, conditioning-branch tensor shapes, and precise meaning of one versus multiple denoising steps during training. What data and optimization schedule were used for the shared decoder in A.17, and how were the two embedding sources mapped into the same conditioning interface?
8 comments
Q2 — Auxiliary-context control
The reviewer is right that this distinction matters. In the current construction, each auxiliary pair is inserted into the original conversation at a random position. Under causal attention, ordinary QA turns that follow the inserted pair can attend to the ground-truth VQ/RGB target tokens. Label masking only removes the loss on user tokens; it does not remove those tokens from the attention context. We thank the reviewer for identifying this.
To isolate this effect, we retrained VLSA with each auxiliary task as an independent example, so that ordinary QA and auxiliary targets never occur in the same attention window. The auxiliary objective itself is unchanged; only its co-occurrence with ordinary QA is removed. The same images, targets, and evaluation protocol were used. Due to time constraints, we were able to run this control only once.
Table Q2: Effect of separating auxiliary targets from ordinary QA.
| Setting | GQA | SQA-I | DocVQA |
|---|---|---|---|
| Full VLSA (auxiliary target in the same conversation) | 65.3 | 77.5 | 75.2 |
| Separate auxiliary example | 65.6 | 77.3 | 75.3 |
The differences are within 0.3 points and have no consistent direction. Thus, removing the pathway by which ordinary QA can see ground-truth targets does not materially change performance, and the gains are not primarily explained by target-context leakage. This control complements the fixed-V utilization test (Table 8; Appendix A.17). We have reported both the control and its single-run limitation in the revised manuscript, and have made the insertion rule explicit in Appendix A.13.
W2/Q3 — Global spatial position in the SA-Perceiver
We agree with the reviewer's mathematical point: stacking high-resolution patch features in spatial order does not, by itself, make cross-attention position-aware, because attention is permutation-invariant over the key set. We would like to clarify the intended meaning of the sentence in Section 3.1 — "we gather information from while preserving their spatial structure according to ." In our design, the low-resolution snapshot is the spatial carrier: its 24×24 queries are position-encoded and provide a coarse spatial anchor. The high-resolution patches are detail carriers: they supply fine-grained content, but they are not intended to carry global crop positions. In the current implementation, the high-resolution keys and values have only CLIP's within-patch position information; the attention mask is a padding/unpad mask, not a correspondence mask. We do not claim fine-grained global assignment of high-resolution patches.
In the case raised by the reviewer, where the low-resolution view cannot distinguish two details, aggregating their high-resolution features into the low-resolution query is harmless (or at least no worse than the naive low-resolution baseline).
To test whether explicit key-side global positions would matter, we trained an ablation variant that adds a parameter-free 2D sincos global position embedding to each high-resolution token. In Table Q3, we observed no material change. Due to time constraints, this ablation was run once.
Table Q3: Ablation with global 2D position embeddings on high-resolution keys.
| Setting | GQA | SQA-I | DocVQA |
|---|---|---|---|
| Current VLSA | 65.3 | 77.5 | 75.2 |
| Ablation: + global key positions | 65.3 | 77.6 | 75.0 |
This indicates that the absence of key-side global positions does not manifest as a material performance degradation on these benchmarks.
In addition, Table 4 (3) vs. (2) isolates the effectiveness of current implementation. Relative to the naive low-resolution baseline (2), row (3) improves ChartQA by 20.5 points (44.8→65.3), DocVQA by 16.4 (45.4→61.8), MME-Perception by 9.2 (1417.3→1426.5), VisWiz by 7.0 (48.9→55.9), and AI2D by 1.9 (67.8→69.7). These are the fine-grained perception tasks for which low-resolution detail is most limiting. The gains show that high-resolution content is effectively injected into the low-resolution query grid; they do not rely on explicit key-side global positions, consistent with the Table Q3 ablation.
Q4 — Reconstruction interface and shared decoder
We have added the following details to Appendix A.2/A.17.
Reconstruction objective. For the input image , a frozen SD3 VAE encodes its low-resolution snapshot into a latent . Following the rectified flow formulation of SD3 [2], we sample Gaussian noise and a continuous timestep coordinate (i.e., ). We quantize to an index and obtain the corresponding and from the SD3 FlowMatchEulerDiscreteScheduler, forming the noisy latent . The SD3 transformer outputs a velocity ; we convert it into the clean-latent estimate and supervise it against with the unweighted MSE in Eq. (3), . Each training example uses one randomly sampled timestep and one transformer forward pass. In the 2-/5-step ablation in Appendix Table 28, we instead unroll denoising updates per iteration and backpropagate through the final loss; this increases training time from 18.9h to 22.1/28.4h with negligible downstream change, so all main results use one step.
[2] P. Esser, S. Kulal, A. Blattmann, R. Entezari, J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer, F. Boesel, et al. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. ICML, 2024.
Conditioning interface. The SA-Perceiver produces and ; is the visual embedding input to the LLM. Epigone is a low-rank MLP with a SiLU activation, applied to each token of ; it has 4096-d input, 256-d hidden, and 4096-d output (in Appendix A.2, Implementation Details, the Epigone sentence reports "4086"; this is a typo for 4096), yielding . The first 576 tokens (from ) form the prompt branch, , which is passed directly to SD3's joint attention without padding; the final token (from ) is projected by a linear head to the pooled embedding. Together, these form the conditioning input in Eq. (4) and match the prompt-plus-pooled interface required by SD3.
Shared decoder (A.17). We use training data from the LLaVA-Next 790K mixture; the 1,000 Q-Bench evaluation images are a separate held-out set and were not used in either VLSA or decoder training. For each training example, we extract one LLaVA-Next embedding () and one VLSA embedding () from the frozen pipelines, pair each embedding with the same target image, and train one SD3 decoder initialized from the official pretrained weights. To map both sources into the same interface, we use a shared token-wise projection . For a source embedding with , this projection yields the prompt sequence . We average over its tokens and apply a shared pooled projection to obtain the pooled vector . Both sources use the same and ; their prompt sequences differ only in length ( vs. 576). In SD3's MMDiT blocks, joint attention is applied over the noisy latent tokens and the prompt sequence , while the pooled vector is supplied separately as SD3's pooled conditioning. Because joint attention accepts prompt sequences of different lengths, we do not align the two sources to a common token count; each is consumed at its native length (with standard batch padding/masking when needed), while always has the same 2048-d shape. All reconstruction-side parameters, including both projections and the denoiser, are shared. The two pairs are weighted equally, , and are optimized with the same flow-matching objective [2] using one denoising step per iteration. Training uses AdamW with a learning rate of , a batch size of 128 examples (256 pairs), a cosine schedule with 3% warmup, zero weight decay, and bf16 precision for 3 epochs without early stopping. The VAE remains frozen, while the denoising transformer and conditioning projections are trainable. At evaluation, the trained decoder is held fixed and applied to both embeddings of these images.
Closing
We appreciate the reviewer’s emphasis on implementation-level clarity. The revised manuscript separates what is measured by the reconstruction objective, what is measured by the fixed-representation utilization test, and what remains an implementation choice.
Dear Reviewer 6UAf,
We thank you for the detailed review and for pointing out the implementation details needed to assess the training procedure and its diagnostics.
W1/Q1 — SSFT sequence construction and training cost
We thank the reviewer for identifying the missing implementation details in the SSFT construction. The reviewer is right that the submitted Table 40 and Section 3.2 reflect an early exploratory configuration and are outdated. In the current efficiency-oriented setting, we adopt the same 7× downsampling configuration in predicting RGB values as the early attempt PVP[1], which introduced a similar RGB pixel-value prediction task. Besides, each conversation receives only one of the two auxiliary tasks (The 10:1 sample-level ratio is the best configuration in Table 27 ablations). We have corrected these descriptions in the revised manuscript and apologize for the confusion.
[1] C. Gou, A. Felemban, F. F. Khan, D. Zhu, J. Cai, H. Rezatofighi, and M. Elhoseiny. How Well Can Vision Language Models See Image Details? arXiv:2408.03940, 2024.
Task construction. The final mixture contains 779,289 released conversations (the nominal 790K training set from LLaVA-Next includes about 15K non-released conversations). Each conversation contains exactly one auxiliary task, and the codebook and pixel tasks are never inserted together. The realized counts are 708,445 VQ samples and 70,844 RGB samples, i.e., 10:1 at the sample level. Table 40 previously illustrated an earlier prompt format that included the image size and required output length. Those hints are not used in the current data, and the revised Table 40 now shows the default 336 configuration with the current single-task prompt.
Target sizes. For a 336×336 snapshot, the selected quarter is 168×168. The VQ-VAE produces 21×21 = 441 codebook indices. The RGB target is the same quarter downsampled 7× to 24×24, i.e., 576 pixels and 1,728 scalar values. This follows the 7× downsampling ratio used in PVP[1] but we predict the dense 24×24 region rather than one coordinate at a time.
Token lengths and truncation. The LLaVA-Next experiments use the LLaMA-3 tokenizer. Stage II uses a 32,768-token context. Each VQ answer tokenizes to approximately 1.2k tokens and the 24×24 RGB answer to approximately 2.9k. For the text portion alone (excluding visual and auxiliary-target tokens), the 790K LLaVA-Next training set has a mean length of 173 tokens and a median of 126; the 90th, 95th, and 99th percentiles are 472, 486, and 527 tokens, respectively, and the longest conversation contains 2,693 tokens. Even under the worst-case, the maximum combined sequence is 576 (visual tokens) + 2,693 (the longest text conversation) + 5,184 (the 24×24 RGB answer: 576 pixels × 9 tokens per bracketed
[r, g, b]triple, including brackets and separators) = 8,453 tokens; no sample exceeds 32,768, so there are zero truncations and more than 24k tokens of headroom.Task ratio. The 10:1 ratio is at the sample level. Because an RGB answer is longer than a VQ answer, the realized token-level ratio is 4.09:1: the VQ samples contribute 834.5M answer tokens and the RGB samples contribute 204.0M. The 10:1 ratio therefore controls how often each task appears, not its loss weight.
Loss normalization. Equation (10) defines the token-summed language-modeling loss over the augmented response; we optimize its token-averaged form over all supervised response tokens, so VQ, RGB, and QA tokens receive equal per-token weight and no task-specific coefficient. The 10:1 ratio controls sampling frequency, not loss weight. The reconstruction objective is separate:
L_all = L + L_rec(Section 3.2); it is added at the objective level and does not enter the token-level normalization ofL.Training time. With 779,289 conversations, one epoch, and the Stage II batch size of 64 (Table 16), the run has 12,176 updates. The baseline average is 3,053 non-padding tokens (2,880 visual + 173 text), while VLSA averages 2,081 tokens (576 visual + 173 text + about 1,332 auxiliary-target tokens under the 10:1 VQ/RGB mix), corresponding to 2.38B and 1.62B non-padding tokens per epoch. The VLSA average includes these auxiliary tokens, and no auxiliary target is truncated; the reduction is therefore not obtained by dropping them. The 32% shorter sequence reduces the quadratic LLM attention cost by about
(2,081/3,053)^2 ≈ 0.46, consistent with the measured Stage II reduction from 30.0h to 14.1h; this measurement includes the single-step LDM/VAE training overhead. Stage I changes only from 5.3h to 4.8h, so the total time falls from 35.3h to 18.9h (Table 41).