- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper targets VAE latents organization for the task of image and video generation vs reconstruction. In particular, it distinguishes between reconstruction-VAE as the most canonical form of autoencoder based on reconstruction fidelity vs representation-VAE as recent formulations that better organize semantic information in latents to improve/facilitate the generative tasks.
Under this picture, the authors ask if the problem of reconstruction-VAEs is the way semantic information is organized, or if semantic information is entirely absent. They answer the question with a small experiment, showing that semantic information is recoverable.
The paper also proposes WeaveAE, an Autoencoder module that starts from a frozen reconstruction-VAE and employs a Vision Foundation Model (VFM) to align the latents towards better semantic organization. Afterwards, it concatenates the two types of latent (reconstruction vs semantic) into a unique representation for the generative model.
The generative component proposed is WeaveDiT, which employs the two latent types of WeaveAE and fuses them together to enable better generative learning. The two components together unify under the framework tamed LatentWeave.
Experimentally, the framework is tested on image and video datasets, showing that the model is suitable both for reconstruction and generative modelling.
Critical strengths
The paper follows a relevant research trend, that is how the latent visual representations should be organized to allow better generative capabilities. The proposal of keeping both semantically aligned and visual fidelity features is interesting, and the authors employ an overall lightweight approach to do so. In particular, I appreciate the idea of keeping the VAE encoder frozen, instead of fine-tuning it to align its latent space to more semantically meaningful information.
Experimentally, Tables 2 and 3 show that the proposed approach leads to good video generations while maintaining reconstructions, which is an encouraging result.
The paper is mostly well written and easy to follow.
Critical weaknesses
The main research question that the authors propose and its answer appear trivial, and as a consequence contribution 1 has little to no impact. Specifically, the question is expressed as: "when VAE latents appear semantically weak, is semantic information largely absent from the representation, or does it remain encoded but poorly exposed by its reconstruction-oriented organization?"
It seems pretty obvious that semantic information must be encoded in the latent somewhere, otherwise the decoder would not be capable of reconstructing the image faithfully. From experiments, it looks like the authors interpret "semantic information" mainly as class discriminative information. Let the autoencoding process be described as , where is the source image, the latent, and the reconstruction. Since attains information from only via , anything the decoder recovers in
must be also in . So the question trivially becomes: "Is class-discriminative information visibile in the reconstructions?"
For any non diverged autoencoder network, the answer is clearly positive.
Alternatively, imagine training an image classifier (e.g. on imagenet classes) using as input not the ground truth images, but reconstructions of a reconstruction-oriented VAE with good reconstruction capability. It is pretty obvious that, if reconstructions are at least decent, the classifier would still be able to learn a meaningful classification pattern.
Hence, the relevant research direction is the one literature already addresses, i.e. how semantic information is organized in the latent space, not if it is present.
What is perhaps more relevant is that the authors propose to keep the VAE encoder frozen, and learn only the semantic mapper. However, from methodology section it appears that the "semantic strand" is learned via a similarity loss over a compressed/project version of the frozen VFM. Therefore, a natural question that emerges is why the semantic mapper is needed at all. More specifically, how does the method perform if compared to a minimal version where is the same, and is obtained directly as a reprojection of VFM features to the correct latent shape?
From a results perspective, some tables are confusing/misleading. In particular, experimental analysis and tables 2/3 focus only on video generation, but the method is presented as dual purpose (image/video). Image results should appear too in the main text, and not only in appendix.
Regarding Table 4 I have concerns on what the results show. Specifically, results of on video show that the table is obtained on the UCF-101 dataset (Table 2). In that table, Wan2.2 VAE (which provides the frozen backbone for the encoder latents), shows a reconstruction on the same dataset of 36.28 PSNR and 4.11 rFVD. From figure 3, it appears like (i.e. Wan2.2 VAE latents) and are the same. But obtain lower reconstruction results 25.28 PSNR 68.27 rFVD in Table 4, than in Table 2.
The logical conclusion is that the problem is not in the latents, but in the decoder path. Table 4 uses the decoder path that has been jointly trained to see both strands, while the "natural decoder" of Wan2.2 VAE (which sees only detail strand) is achieves better PSNR than .
Therefore, the authors conclusion at line 431: "These results show that recovered semantic structure complements, rather than
replaces, the reconstruction-oriented information inherited from the pretrained VAE." should be clarified. The result is true only on a decoder that has been trained to see both components, and is therefore quite natural that masking some features degrades performances.
Questions for authors
Minor Comment: The Self-REPA component is named in abstract and specified in methodology, but it is not listed among contributions or named whatsoever in the introduction.
Clarification Request The proposed method employs both semantic and reconstruction latents, establishing therefore a tradeoff between reconstruction oriented VAEs (which perform best in visual fidelity) and semantic oriented VAEs (which make generation task easier). The questions that emerge from this approach and could better motivate it/enhance its significance are:
- Can a VAE be “optimal” at both reconstruction and semantic tasks, or is some sort of specialization inevitable?
- In the latter case, what is the best achievable tradeoff? In that sense, can the proposed method be tuned to prioritize a task versus the other, i.e. "selecting" the desired tradeoff?
- For practitioners, if the interest is reconstruction OR generation, what are the benefits of the proposed method w.r.t. selecting a VAE that directly specializes on latents suitable for the desired task?
5 comments
📝[W3 & Minor Comment] Main-Text Image Results & Introducing Self-REPA
Thank you for these suggestions. Both can be addressed through targeted presentation changes.
💡1. Presenting image results alongside video results. Due to the submission page limit, the ImageNet-1K and MS-COCO results were included in Appendix Tables 8 and 9. We will incorporate these results into the main Experiments section as space permits.
💡2. Introducing Self-REPA earlier. We will briefly introduce Self-REPA in the Introduction, explaining how it reuses the tokenizer’s semantic features for generative alignment without an external VFM. This will make its role clear before the detailed methodology.
📝[W4] What Table 4 Shows: Strand Dependence in Joint Decoding
The reviewer is correct that Table 4 evaluates a decoder trained to use both strands. Removing one strand measures this decoder’s dependence on that input; it does not establish that the frozen VAE latent lacks the information needed for reconstruction. The strong reconstruction performance of the original Wan2.2 decoder is consistent with this distinction. We will clarify the statement at Line 431 accordingly.
To further examine how the jointly trained decoder uses the two strands, we compare WeaveAE with direct-concatenation baselines under the same UCF-101 evaluation setting. All configurations use the ST-ViT decoder architecture. The entries below report reconstruction PSNR.
| Tokenizer | Channels | Both Strands | Detail Only | Semantic Only |
|---|---|---|---|---|
| Raw VFM concatenation (concat-1200) | 1200 | 31.19 | 30.05 | 11.16 |
| Direct VFM reprojection (com96) | 96 | 31.17 | 30.26 | 11.13 |
| VAE-latent Semantic Mapper (WeaveAE, ours) | 96 | 32.60 | 24.21 | 21.91 |
💡1. Greater use of the semantic strand. Direct-concatenation decoders rely predominantly on : removing causes a relatively small PSNR drop, whereas removing reduces it to approximately 11.1 dB. For WeaveAE, removing either strand substantially degrades reconstruction, consistent with its detail-masking design encouraging the decoder to use .
💡2. Complementary roles within the joint decoder. The paper’s visualizations show that semantic-only decoding preserves coarse structure and layout, while detail-only decoding retains local texture but exhibits degraded global coherence. Together with the PSNR improvement from using both strands, these observations support complementary roles within the learned decoder—not an information deficit in the original VAE latent.
Together, these results show that the two strands play complementary roles in WeaveAE’s jointly trained decoder, while remaining consistent with the original VAE latent supporting faithful reconstruction through its pretrained decoder.
📝[Clarification Request] Reconstruction–Semantic Tradeoffs and Single-Task Benefits
💡1. Is specialization inevitable?
Not necessarily. Reconstruction and semantic learning favor different representation properties: reconstruction benefits from sensitivity to local detail, whereas semantic learning often encourages invariance to such variations. This tension does not imply fundamental incompatibility. WeaveAE addresses these competing demands by keeping the pretrained VAE itself frozen and introducing lightweight modules to organize its latents into semantic and detail strands. The original VAE representation is retained, while semantic structure becomes more accessible through the additional strand.
💡2. Can the tradeoff be tuned?
Yes, through training settings rather than inference-time adjustment. The design provides several training controls without modifying the pretrained VAE. During Stage 1, the alignment and semantic-preservation weights govern semantic organization. During Stage 2, perceptual/adversarial weights and the detail-masking ratio govern decoder training and its use of the two strands. These settings allow different training configurations, rather than inference-time adjustment. Our experiments demonstrate improved perceptual reconstruction at the selected configuration; a systematic tradeoff curve has not been evaluated.
💡3. What are the benefits for a single task?
- Generation-only: LatentWeave improves generation quality over the native VAE baseline, reducing UCF-101 FVD from 212.2 to 135.5 and ImageNet-1K gFID from 10.62 to 5.26 without CFG. Its 96-channel representation is also 12× narrower than RAE’s 1152-channel representation. After tokenizer training, neither encoding nor generative training requires an external VFM. These benefits remain relevant even when reconstruction is not the primary objective.
- Reconstruction-only: WeaveAE improves perceptual reconstruction metrics, including ImageNet-1K rFID from 0.76 to 0.15, Kinetics-600 rFVD from 1.49 to 1.16, and UCF-101 rFVD from 4.11 to 3.11, while trading off some pixel-level accuracy. Its ST-ViT decoder also achieves 11.6× lower decoding latency in our reported benchmark. It therefore offers benefits when perceptual fidelity and decoding speed are priorities. Because these capabilities are added without updating the pretrained VAE, users retain its original encoding and decoding paths, including the original decoder when higher PSNR is preferred.
📝[W1] Research Motivation and the Role of Probing
💡1. Semantic preservation versus semantic accessibility: We agree with the reviewer’s reasoning based on x→z→x^: semantic information recoverable from the reconstruction must also be recoverable from the latent. The presence of such information is therefore expected, rather than itself a surprising finding. Our motivation concerns its accessibility: information sufficient for reconstruction does not necessarily form a representation that is readily usable for semantic readout or generative modeling.
💡2. Probing motivates reorganization of the frozen VAE latent: Figure 2 illustrates this distinction: on ImageNet-1K, a linear probe achieves 4.33% accuracy on frozen VAE latents, whereas a two-block ViT probe reaches 43.6%. This gap motivates learning a more accessible representation; it is not intended as evidence that semantic information was previously absent. The contribution is WeaveAE’s approach to this reorganization: deriving a compact semantic strand directly from frozen VAE latents while retaining the native latent as the detail strand. Semantic supervision is used during tokenizer training, but the resulting representation requires neither encoder modification nor a VFM during subsequent encoding or generation.
📝[W2] Why Learn a Semantic Mapper Instead of Directly Reprojecting VFM Features?
Following the reviewer’s suggestion, we evaluated direct-feature baselines that retain the frozen Wan2.2 detail latent zd and construct the semantic strand from frozen SigLIP2 features aligned to the causal latent grid. We consider two variants: concat-1200 appends the raw 1152-channel features, while com96 projects them to 48 channels, matching WeaveAE’s 96-channel latent shape and ST-ViT decoder architecture. All configurations use the same UCF-101 Latte-XL training and evaluation protocol.
Additional feature extraction costs are reported relative to the shared Wan2.2 encoder.
💡1. The mapper removes the separate VFM encoding pass. Direct reprojection requires SigLIP2 features when encoding a new clip. WeaveAE instead computes its semantic strand from the compressed 5×16×16 VAE token grid and discards SigLIP2 after tokenizer training. Its semantic extraction cost is 371.9 GFLOPs per clip, compared with 3747.8 GFLOPs for the 17-frame SigLIP2 pass—approximately 10.1× lower.
💡2. Direct reprojection helps, but does not match the learned mapper. The unaligned 96-channel control provides no improvement over the native 48-channel baseline at 100K steps. Raw concatenation performs poorly under the standard unweighted velocity-MSE objective; its VFM channels occupy 96% of the latent dimensions, creating a potential imbalance between the semantic and detail objectives. Compressing the VFM features substantially improves performance, reaching an FVD of 294.2.
The most direct comparison is therefore com96 versus WeaveAE. With the same latent dimensionality, decoder architecture, and generator protocol, WeaveAE reduces FVD from 740.8 to 472.3 at 50K steps and from 294.2 to 214.2 at 100K steps. These results support learning the semantic strand from the VAE latent rather than directly appending projected VFM features: it achieves better generation quality in this setting while avoiding a separate VFM encoding pass.