acceptodds
Withdrawn from ICLR 2027

LatentWeave: Are Semantics Missing from VAE Latents—or Merely Implicit?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual encoders define the representation spaces underlying visual reconstruction, understanding, and generation. Reconstruction-oriented VAEs provide compact representations with strong visual fidelity, yet their latents are shaped primarily by reconstruction and exhibit weak semantic organization. Recent approaches address this limitation by replacing VAE encoders with pretrained representation encoders. We revisit whether such replacement is necessary and find that substantial semantic information remains recoverable from pretrained VAE latents despite being poorly exposed in their native organization. Motivated by this observation, we introduce LatentWeave, a unified framework for constructing and generatively modeling semantically organized visual latents from pretrained VAE representations. It comprises WeaveAE, which organizes complementary detail and semantic strands, and WeaveDiT, which explicitly exploits this dual-strand structure during generative modeling. We further introduce Self-REPA, which reuses WeaveAE’s semantic representation as an internal alignment target to improve generative learning without introducing an external vision encoder during generator training. Across image and video benchmarks, LatentWeave improves semantic accessibility and generative learning efficiency while preserving reconstruction fidelity. On UCF-101, WeaveAE achieves an rFVD of 3.11 for reconstruction, while WeaveDiT achieves an FVD of 135.49 for class-conditional generation. Our results establish latent reorganization from pretrained VAE representations as a practical paradigm for unified visual representation learning and efficient generative modeling.

Primary area:
generative models
Keywords:
Video Generative Models, Representation Learning, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UbfiOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets VAE latents organization for the task of image and video generation vs reconstruction. In particular, it distinguishes between reconstruction-VAE as the most canonical form of autoencoder based on reconstruction fidelity vs representation-VAE as recent formulations that better organize semantic information in latents to improve/facilitate the generative tasks.

Under this picture, the authors ask if the problem of reconstruction-VAEs is the way semantic information is organized, or if semantic information is entirely absent. They answer the question with a small experiment, showing that semantic information is recoverable.

The paper also proposes WeaveAE, an Autoencoder module that starts from a frozen reconstruction-VAE and employs a Vision Foundation Model (VFM) to align the latents towards better semantic organization. Afterwards, it concatenates the two types of latent (reconstruction vs semantic) into a unique representation for the generative model.

The generative component proposed is WeaveDiT, which employs the two latent types of WeaveAE and fuses them together to enable better generative learning. The two components together unify under the framework tamed LatentWeave.

Experimentally, the framework is tested on image and video datasets, showing that the model is suitable both for reconstruction and generative modelling.

Critical strengths

The paper follows a relevant research trend, that is how the latent visual representations should be organized to allow better generative capabilities. The proposal of keeping both semantically aligned and visual fidelity features is interesting, and the authors employ an overall lightweight approach to do so. In particular, I appreciate the idea of keeping the VAE encoder frozen, instead of fine-tuning it to align its latent space to more semantically meaningful information.

Experimentally, Tables 2 and 3 show that the proposed approach leads to good video generations while maintaining reconstructions, which is an encouraging result.

The paper is mostly well written and easy to follow.

Critical weaknesses

The main research question that the authors propose and its answer appear trivial, and as a consequence contribution 1 has little to no impact. Specifically, the question is expressed as: "when VAE latents appear semantically weak, is semantic information largely absent from the representation, or does it remain encoded but poorly exposed by its reconstruction-oriented organization?"

It seems pretty obvious that semantic information must be encoded in the latent somewhere, otherwise the decoder would not be capable of reconstructing the image faithfully. From experiments, it looks like the authors interpret "semantic information" mainly as class discriminative information. Let the autoencoding process be described as X^=D(z);z=E(X)\hat{X} = D(z); z = E(X), where XX is the source image, zz the latent, and X^\hat{X} the reconstruction. Since X^\hat{X} attains information from XX only via zz, anything the decoder recovers in X^\hat{X} must be also in zz. So the question trivially becomes: "Is class-discriminative information visibile in the reconstructions?"
For any non diverged autoencoder network, the answer is clearly positive.
Alternatively, imagine training an image classifier (e.g. on imagenet classes) using as input not the ground truth images, but reconstructions of a reconstruction-oriented VAE with good reconstruction capability. It is pretty obvious that, if reconstructions are at least decent, the classifier would still be able to learn a meaningful classification pattern.
Hence, the relevant research direction is the one literature already addresses, i.e. how semantic information is organized in the latent space, not if it is present.

What is perhaps more relevant is that the authors propose to keep the VAE encoder frozen, and learn only the semantic mapper. However, from methodology section it appears that the "semantic strand" is learned via a similarity loss over a compressed/project version of the frozen VFM. Therefore, a natural question that emerges is why the semantic mapper is needed at all. More specifically, how does the method perform if compared to a minimal version where zdz_d is the same, and zsz_s is obtained directly as a reprojection of VFM features to the correct latent shape?

From a results perspective, some tables are confusing/misleading. In particular, experimental analysis and tables 2/3 focus only on video generation, but the method is presented as dual purpose (image/video). Image results should appear too in the main text, and not only in appendix.

Regarding Table 4 I have concerns on what the results show. Specifically, results of zuz_u on video show that the table is obtained on the UCF-101 dataset (Table 2). In that table, Wan2.2 VAE (which provides the frozen backbone for the encoder latents), shows a reconstruction on the same dataset of 36.28 PSNR and 4.11 rFVD. From figure 3, it appears like zvaez_\text{vae} (i.e. Wan2.2 VAE latents) and zdz_d are the same. But zdz_d obtain lower reconstruction results 25.28 PSNR 68.27 rFVD in Table 4, than zvaez_\text{vae} in Table 2.
The logical conclusion is that the problem is not in the latents, but in the decoder path. Table 4 uses the decoder path that has been jointly trained to see both strands, while the "natural decoder" of Wan2.2 VAE (which sees only detail strand) is achieves better PSNR than zuz_u.
Therefore, the authors conclusion at line 431: "These results show that recovered semantic structure complements, rather than replaces, the reconstruction-oriented information inherited from the pretrained VAE." should be clarified. The result is true only on a decoder that has been trained to see both components, and is therefore quite natural that masking some features degrades performances.

Questions for authors

Minor Comment: The Self-REPA component is named in abstract and specified in methodology, but it is not listed among contributions or named whatsoever in the introduction.

Clarification Request The proposed method employs both semantic and reconstruction latents, establishing therefore a tradeoff between reconstruction oriented VAEs (which perform best in visual fidelity) and semantic oriented VAEs (which make generation task easier). The questions that emerge from this approach and could better motivate it/enhance its significance are:

  • Can a VAE be “optimal” at both reconstruction and semantic tasks, or is some sort of specialization inevitable?
  • In the latter case, what is the best achievable tradeoff? In that sense, can the proposed method be tuned to prioritize a task versus the other, i.e. "selecting" the desired tradeoff?
  • For practitioners, if the interest is reconstruction OR generation, what are the benefits of the proposed method w.r.t. selecting a VAE that directly specializes on latents suitable for the desired task?
5 comments

📝[W1] Research Motivation and the Role of Probing

💡1. Semantic preservation versus semantic accessibility: We agree with the reviewer’s reasoning based on x→z→x^x \rightarrow z \rightarrow \hat{x}: semantic information recoverable from the reconstruction must also be recoverable from the latent. The presence of such information is therefore expected, rather than itself a surprising finding. Our motivation concerns its accessibility: information sufficient for reconstruction does not necessarily form a representation that is readily usable for semantic readout or generative modeling.

💡2. Probing motivates reorganization of the frozen VAE latent: Figure 2 illustrates this distinction: on ImageNet-1K, a linear probe achieves 4.33% accuracy on frozen VAE latents, whereas a two-block ViT probe reaches 43.6%. This gap motivates learning a more accessible representation; it is not intended as evidence that semantic information was previously absent. The contribution is WeaveAE’s approach to this reorganization: deriving a compact semantic strand directly from frozen VAE latents while retaining the native latent as the detail strand. Semantic supervision is used during tokenizer training, but the resulting representation requires neither encoder modification nor a VFM during subsequent encoding or generation.


📝[W2] Why Learn a Semantic Mapper Instead of Directly Reprojecting VFM Features?

Following the reviewer’s suggestion, we evaluated direct-feature baselines that retain the frozen Wan2.2 detail latent zdz_d and construct the semantic strand from frozen SigLIP2 features aligned to the causal latent grid. We consider two variants: concat-1200 appends the raw 1152-channel features, while com96 projects them to 48 channels, matching WeaveAE’s 96-channel latent shape and ST-ViT decoder architecture. All configurations use the same UCF-101 Latte-XL training and evaluation protocol.

Latent ConstructionChannelsVFM Required for EncodingAdditional Feature Extraction GFLOPs/clip↓FVD@50K↓FVD@100K↓
Wan2.2 VAE (native baseline)48✗—993.4330.7
Wan 96ch (w/o align baseline)96✗371.9770.0344.7
Raw VFM concatenation (concat-1200)1200✓3747.82855.01896.0
Direct VFM reprojection (com96; reviewer-suggested baseline)96✓3747.8740.8294.2
VAE-latent Semantic Mapper (WeaveAE, ours)96✗371.9472.3214.2

Additional feature extraction costs are reported relative to the shared Wan2.2 encoder.

💡1. The mapper removes the separate VFM encoding pass. Direct reprojection requires SigLIP2 features when encoding a new clip. WeaveAE instead computes its semantic strand from the compressed 5×16×165 \times 16 \times 16 VAE token grid and discards SigLIP2 after tokenizer training. Its semantic extraction cost is 371.9 GFLOPs per clip, compared with 3747.8 GFLOPs for the 17-frame SigLIP2 pass—approximately 10.1× lower.

💡2. Direct reprojection helps, but does not match the learned mapper. The unaligned 96-channel control provides no improvement over the native 48-channel baseline at 100K steps. Raw concatenation performs poorly under the standard unweighted velocity-MSE objective; its VFM channels occupy 96% of the latent dimensions, creating a potential imbalance between the semantic and detail objectives. Compressing the VFM features substantially improves performance, reaching an FVD of 294.2.

The most direct comparison is therefore com96 versus WeaveAE. With the same latent dimensionality, decoder architecture, and generator protocol, WeaveAE reduces FVD from 740.8 to 472.3 at 50K steps and from 294.2 to 214.2 at 100K steps. These results support learning the semantic strand from the VAE latent rather than directly appending projected VFM features: it achieves better generation quality in this setting while avoiding a separate VFM encoding pass.


📝[W3 & Minor Comment] Main-Text Image Results & Introducing Self-REPA

Thank you for these suggestions. Both can be addressed through targeted presentation changes.

💡1. Presenting image results alongside video results. Due to the submission page limit, the ImageNet-1K and MS-COCO results were included in Appendix Tables 8 and 9. We will incorporate these results into the main Experiments section as space permits.

💡2. Introducing Self-REPA earlier. We will briefly introduce Self-REPA in the Introduction, explaining how it reuses the tokenizer’s semantic features for generative alignment without an external VFM. This will make its role clear before the detailed methodology.


📝[W4] What Table 4 Shows: Strand Dependence in Joint Decoding

The reviewer is correct that Table 4 evaluates a decoder trained to use both strands. Removing one strand measures this decoder’s dependence on that input; it does not establish that the frozen VAE latent lacks the information needed for reconstruction. The strong reconstruction performance of the original Wan2.2 decoder is consistent with this distinction. We will clarify the statement at Line 431 accordingly.

To further examine how the jointly trained decoder uses the two strands, we compare WeaveAE with direct-concatenation baselines under the same UCF-101 evaluation setting. All configurations use the ST-ViT decoder architecture. The entries below report reconstruction PSNR.

TokenizerChannelsBoth Strands (zd,zs)(z_d, z_s)Detail Only (zd,0)(z_d, 0)Semantic Only (0,zs)(0, z_s)
Raw VFM concatenation (concat-1200)120031.1930.0511.16
Direct VFM reprojection (com96)9631.1730.2611.13
VAE-latent Semantic Mapper (WeaveAE, ours)9632.6024.2121.91

💡1. Greater use of the semantic strand. Direct-concatenation decoders rely predominantly on zdz_d: removing zsz_s causes a relatively small PSNR drop, whereas removing zdz_d reduces it to approximately 11.1 dB. For WeaveAE, removing either strand substantially degrades reconstruction, consistent with its detail-masking design encouraging the decoder to use zsz_s.

💡2. Complementary roles within the joint decoder. The paper’s visualizations show that semantic-only decoding preserves coarse structure and layout, while detail-only decoding retains local texture but exhibits degraded global coherence. Together with the PSNR improvement from using both strands, these observations support complementary roles within the learned decoder—not an information deficit in the original VAE latent.

Together, these results show that the two strands play complementary roles in WeaveAE’s jointly trained decoder, while remaining consistent with the original VAE latent supporting faithful reconstruction through its pretrained decoder.


📝[Clarification Request] Reconstruction–Semantic Tradeoffs and Single-Task Benefits

💡1. Is specialization inevitable?

Not necessarily. Reconstruction and semantic learning favor different representation properties: reconstruction benefits from sensitivity to local detail, whereas semantic learning often encourages invariance to such variations. This tension does not imply fundamental incompatibility. WeaveAE addresses these competing demands by keeping the pretrained VAE itself frozen and introducing lightweight modules to organize its latents into semantic and detail strands. The original VAE representation is retained, while semantic structure becomes more accessible through the additional strand.

💡2. Can the tradeoff be tuned?

Yes, through training settings rather than inference-time adjustment. The design provides several training controls without modifying the pretrained VAE. During Stage 1, the alignment and semantic-preservation weights govern semantic organization. During Stage 2, perceptual/adversarial weights and the detail-masking ratio govern decoder training and its use of the two strands. These settings allow different training configurations, rather than inference-time adjustment. Our experiments demonstrate improved perceptual reconstruction at the selected configuration; a systematic tradeoff curve has not been evaluated.

💡3. What are the benefits for a single task?

  • Generation-only: LatentWeave improves generation quality over the native VAE baseline, reducing UCF-101 FVD from 212.2 to 135.5 and ImageNet-1K gFID from 10.62 to 5.26 without CFG. Its 96-channel representation is also 12× narrower than RAE’s 1152-channel representation. After tokenizer training, neither encoding nor generative training requires an external VFM. These benefits remain relevant even when reconstruction is not the primary objective.
  • Reconstruction-only: WeaveAE improves perceptual reconstruction metrics, including ImageNet-1K rFID from 0.76 to 0.15, Kinetics-600 rFVD from 1.49 to 1.16, and UCF-101 rFVD from 4.11 to 3.11, while trading off some pixel-level accuracy. Its ST-ViT decoder also achieves 11.6× lower decoding latency in our reported benchmark. It therefore offers benefits when perceptual fidelity and decoding speed are priorities. Because these capabilities are added without updating the pretrained VAE, users retain its original encoding and decoding paths, including the original decoder when higher PSNR is preferred.
Reviewer_uE14OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether pretrained VAE latents contain semantic information that is difficult to access. The authors show that nonlinear probes can recover more semantic information than linear probes. They proposed LatentWeave, which combines three components: WeaveAE to organize semantic and visual detail features, WeaveDiT to model these features for generation, and Self-REPA to improve generative training through semantic alignment. Experiments cover image and video datasets, including ImageNet-1K, Kinetics-600, and UCF-101. Results show improved linear probing accuracy and better image and video generation. The authors also report faster training convergence.

Critical strengths
  1. The paper shows that pretrained VAE latents contain semantic information that can be recovered with nonlinear transformations. This is an interesting observation. The proposed approach improves semantic accessibility without replacing the pretrained VAE encoder. It offers a useful alternative to methods based on representation encoders.

  2. The method improves semantic probing accuracy and generative quality. This is achieved maintaining competitive reconstruction performance. The reported gains in training efficiency are substantial (sometimes difficult to justify like the results from Table 3 where the improvement wrt SoTA is huge, however this trend is not allowed in the other tables). The experiments cover several datasets and include ablations of the main components. This supports the practical value of the approach.

Critical weaknesses
  1. The method combines existing ideas, including semantic alignment with pretrained vision models, dual-stream architectures, and representation alignment for diffusion models. The main novelty seems to be their combination. The paper does not clearly demonstrate what is fundamentally new compared to recent related methods. This weakens the originality of the contribution.

  2. The semantic recovery relies on supervision from SigLIP2. Thus, it is unclear how much semantic information is actually recovered from the VAE latents or the learned through teacher supervision. The comparison also changes several components, including the latent representation, decoder, and generative architecture. Although some ablations are provided, they do not fully isolate these factors or establish comparable computational costs. This makes it difficult to assess the source of the reported improvements as well as the training efficiency.

Questions for authors
  1. How can you distinguish semantic information already present in the VAE latents from information learned through SigLIP2 supervision? Can they provide experiments with weaker or different semantic teachers to support their main claim?

  2. What are the main technical differences between LatentWeave and existing approaches using semantic alignment and dual-stream generative architectures? Which component provides a really new contribution?

  3. Are the reported improvements in generation quality and training efficiency maintained when controlling for the decoder, latent dimensionality, model parameters, and total training FLOPs? Can you provide a more controlled comparison?

  4. How much of the improvement comes from the semantic strand, WeaveDiT architecture, and Self-REPA independently? Can you quantify their individual contributions under identical training conditions?

5 comments

📝[W1 & Q2] Technical Novelty Beyond Combining Existing Ideas

LatentWeave’s distinguishing design is to derive both semantic features and reconstruction-oriented detail from a frozen VAE encoder, and use this shared representation for generation and alignment. The VFM provides supervision during tokenizer training but is not required for subsequent encoding, generative training, or inference. This differs from approaches that modify the VAE encoder, use a VFM as part of the tokenizer, or introduce an external teacher during diffusion training.

ApproachRepresentative methodsWhere semantic information enters
VAE-encoder alignmentVA-VAE, REPA-ESemantic supervision modifies the VAE encoder.
VFM-based tokenizationRAE, SVGA VFM supplies features as part of the encoding pipeline.
External diffusion alignmentREPAAn external VFM supplies targets for diffusion training.
LatentWeave (ours)WeaveAE + WeaveDiT + Self-REPASemantic features are derived from frozen VAE latents and reused for generation and alignment.

💡1. WeaveAE exposes semantic structure while retaining the native VAE latent: Its Semantic Mapper operates on compressed VAE tokens, while the detail strand retains the original VAE representation. For a 17-frame clip, the mapper requires 371.9 GFLOPs, compared with 3747.8 GFLOPs for the SigLIP2 pass—a roughly tenfold reduction in this additional encoding cost. The semantic-preservation loss, Lkeep\mathcal{L}_{\text{keep}}, further maintains semantic accessibility through the 48-channel bottleneck: removing it reduces ImageNet-1K linear probing accuracy on zsz_s from 65.68% to 15.20% (Table 5).

💡2. WeaveDiT models complementary channel subspaces on a shared visual grid: Unlike text–image dual-stream designs that process separate token sequences, WeaveDiT assigns dedicated branches to the spatially co-located semantic and detail strands, with lightweight cross-strand interaction. Our controlled comparisons in Q3 show that its benefits extend to reconstruction-only latents, while its combination with WeaveAE achieves the best generation quality among the tested configurations.

💡3. Self-REPA reuses the tokenizer’s semantic features as alignment targets: Rather than requiring a separate VFM forward pass to obtain alignment targets, Self-REPA uses WeaveAE’s internal hsh_s, which already matches the latent grid. The additional alignment computation is 0.01 TFLOPs per clip, approximately 0.2% of the reported generator training compute.

These designs form a connected pipeline: WeaveAE makes semantic structure accessible from frozen VAE latents, WeaveDiT models the resulting semantic and detail strands, and Self-REPA reuses their semantic features to guide generative training. The contribution lies in this specific representation and its coordinated use across tokenization, generation, and alignment.

📝[W2 (Part 1)] Distinguishing Latent Information from Teacher Supervision

The VAE latent supplies the visual input; SigLIP2 supplies supervision for learning an accessible representation. These roles are different. Our evidence shows that semantic prediction is possible without VFM supervision, while WeaveAE uses teacher supervision to improve semantic accessibility.

💡1. Semantic prediction is possible without VFM supervision: In Fig. 2, both linear and nonlinear probes are trained on frozen VAE latents using classification labels alone. Replacing the linear probe with a shallow two-block ViT raises accuracy from 4.33% to 43.6% on ImageNet-1K and from 2.8% to 35.2% on Kinetics-700. These results show that the latents support substantially stronger semantic prediction than their linear readout suggests, independently of SigLIP2 supervision.

💡2. Teacher supervision helps organize the representation: During tokenizer training, SigLIP2 provides a representation target for learning the semantic mapper. After training, the teacher is discarded, and the semantic strand is computed solely as zs=Msem(zvae)z_s = M_{\mathrm{sem}}(z_{\mathrm{vae}}). Thus, teacher supervision determines how the mapper extracts and organizes features, while each sample’s visual features are obtained entirely from its VAE latent. The resulting representation supports a stronger linear readout, as summarized below.

Representation and ReadoutVFM SupervisionImageNet-1K Top-1↑Kinetics-700 Top-1↑UCF-101 Top-1↑
Native VAE baseline: linear probe✗4.33%2.8%18.51%
Native VAE baseline: two-block ViT probe✗43.6%35.2%—
WeaveAE (ours): linear probe on zuz_u✓ SigLIP264.97%—72.41%

All probes use classification labels for training. No VFM is required at evaluation time. The alternative-teacher experiments in Q1 further test whether the gains depend specifically on SigLIP2.


📝[W2 (Part 2) & Q3] Controlling Representation, Decoder Architecture, and Training Compute

We conducted additional comparisons under the same UCF-101 training and evaluation protocol. All 96-channel configurations use the same ST-ViT decoder architecture, with 24 blocks and hidden dimension 1024. Within this group, we compare representations using the same Latte-XL generator, and compare generators while holding the representation fixed. We also report generator training FLOPs to make the computational differences explicit.

LatentChannelsDecoder architectureGeneratorGenerator paramsTraining TFLOPs/clipFVD@50K↓FVD@100K↓
Wan 48ch (native baseline)48Wan 3D-ConvLatte-XL675M3.68993.0331.0
Wan 96ch (w/o align)96ST-ViTLatte-XL675M3.68770.0344.7
Wan 96ch (w/o align)96ST-ViTWeaveDiT685M4.34538.9310.7
WeaveAE (ours)96ST-ViTLatte-XL675M3.68472.3214.2
WeaveAE (ours)96ST-ViTWeaveDiT685M4.34314.0192.0
LatentWeave (ours, full model)96ST-ViTWeaveDiT + Self-REPA685M4.35276.8168.7

💡1. A wider latent alone does not explain the gains. The unaligned 96-channel control does not improve 100K-step FVD over the original 48-channel baseline: 344.7 versus 331.0. Since these two configurations also use different decoders, this comparison evaluates the wider latent together with the decoder change, rather than channel capacity in isolation.

💡2. WeaveAE improves quality with channel count, decoder architecture, and generator compute held fixed. Among the 96-channel configurations using ST-ViT decoders and Latte-XL, WeaveAE achieves an FVD of 214.2, compared with 344.7 for the unaligned control. Both use approximately 675M generator parameters and 3.68 TFLOPs per clip. This supports the benefit of the learned representation beyond increased latent width or a different generator.

💡3. WeaveDiT and Self-REPA provide further gains on the same representation. With WeaveAE fixed, replacing Latte-XL with WeaveDiT reduces FVD from 214.2 to 192.0. Adding Self-REPA further reduces it to 168.7, with an additional 0.01 TFLOPs per clip. WeaveDiT also improves the unaligned 96-channel control, from 344.7 to 310.7, showing that its benefit is not restricted to WeaveAE.

💡4. The quality gains remain substantial after accounting for generator training compute. The full model uses 4.35 TFLOPs per clip, approximately 18% more than Latte-XL, while reducing 100K-step FVD from 331.0 to 168.7. It also reaches an FVD of 276.8 at 50K steps, surpassing the original baseline’s 100K-step result. Under the same batch size and FLOP accounting, this checkpoint uses approximately 59% of the baseline’s total generator training FLOPs.

Together, these comparisons show improvements from the learned representation, the generator architecture, and the alignment objective under explicitly stated controls. The final result reflects their combined effect; the individual differences above are conditional comparisons rather than independent additive contributions.


Reviewer_RL7yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LatentWeave, which extracts semantic features from pretrained VAE latents and combines them with reconstruction features for image and video generation. It also introduces a dual-strand DiT architecture to improve generation quality and training efficiency.

Critical strengths
  • The method shows clear improvements in generation quality and training convergence over VAE-based baselines.

  • The experiments cover both image and video generation, with reasonable ablations of the proposed components.

Critical weaknesses
  • Limited novelty in the main motivation. It is hardly surprising that VAE latents contain semantic information, given that they preserve enough information to reconstruct the original images. The probing experiments largely confirm something already expected.

  • The proposed method seems unnecessarily complicated. Adding semantic features to reconstruction-oriented latents is a fairly natural idea. The paper does not convincingly show why the Semantic Mapper and dual-strand design are needed over simpler alternatives.

  • Limited experimental validation. The video experiments focus on small datasets, low resolutions, and short clips, making it hard to judge whether the method would scale to realistic video generation settings.

Questions for authors
  • What is the actual insight behind the probing experiments? One could decode the VAE latents and apply a pretrained classifier or representation encoder to recover semantics. Why should the presence of semantic information in VAE latents be considered a surprising finding?

  • Why not simply concatenate VAE and RAE features? How would this straightforward baseline compare with the proposed approach? Is the additional complexity of the Semantic Mapper really justified?

  • How much of the improvement comes from the proposed architecture rather than additional semantic supervision and latent capacity? The comparison with the original VAE latent does not fully isolate these factors.

5 comments

📝[W1 & Q1] Motivation & Actual Insight of Probing

💡1. Full agreement on semantic preservation: We fully agree with Reviewer RL7y that a latent space trained for high-fidelity reconstruction is inherently aligned with the pixel space. Because pixel space faithfully retains visual semantics, decoding the latent back to pixels and applying an off-the-shelf classifier or VFM (decode + classifier) naturally recovers these semantics. Its presence is an expected outcome of high-fidelity autoencoding rather than a surprising discovery. Our original phrasing ("semantic absence vs. implicit/recoverable semantics") inappropriately framed this as an unexpected discovery. We will sharpen the wording to emphasize reorganizing semantic information retained in frozen VAE latents into a more accessible representation for generative modeling.

💡2. The diagnostic insight of Fig. 2 (pixel-space recoverability vs. direct linear accessibility): While decode + classifier demonstrates pixel-space recoverability via a deep causal decoder and vision backbone, diffusion transformers (DiTs) ingest latents directly at noisy timesteps ztz_t through a single linear PatchEmbed layer, without access to pixel decoding during generation. Probing frozen zvaez_{\text{vae}} directly without pixel decoding yields near-chance linear readout, whereas a shallow 2-block ViT probe achieves a +32.4 to +39.3 pp jump across FLUX, MiniMax, and Wan2.2 VAEs (Fig. 2). This confirms that the core bottleneck is entangled latent geometry rather than information absence, providing the direct motivation for our single-encoder latent reorganization:

Readout PathPixel Decode?Inference Enc.IN-1K Top-1↑Video Top-1↑
Decode -> SigLIP2 (Linear, ref.)✓VAE + VFM78.43%81.23% (UCF)
Frozen zvaez_{\text{vae}} (48c, Linear)✗VAE only4.33%18.51% (UCF) / 2.8% (K700)
Frozen zvaez_{\text{vae}} (48c, 2-block ViT)✗VAE only43.6%35.2% (K700, Fig. 2)
WeaveAE zuz_u (96c, Linear, ours)✗VAE only64.97%72.41% (UCF)

📝[W2 & Q2] Why Not Direct Concatenation?

We evaluated concatenating frozen Wan 48ch with SigLIP2 (1152ch) features, both raw concatenation (concat-1200) and compressed concatenation (com96)). The latter matches WeaveAE’s 96-channel latent shape and ST-ViT decoder architecture. All representation comparisons use the same Latte-XL protocol on UCF-101.

💡1. Raw concatenation performs poorly: Appending 1152-channel SigLIP2 features to the 48-channel VAE latent gives an FVD of 1896.0 at 100K steps, compared with 214.2 for WeaveAE. Under uniform channel weighting, the semantic features occupy 96% of the loss dimensions, creating a potential imbalance between the semantic and detail strands.

💡2. Compressed concatenation improves quality, but remains less effective and more costly to encode: Compressing SigLIP2 features to 48 channels gives com96 the same 96-channel budget as WeaveAE. Nevertheless, WeaveAE achieves lower FVD at both checkpoints, reaching 214.2 at 100K steps compared with 294.2 for com96. Encoding also requires 4.34 rather than 7.71 TFLOPs per clip. The mapper therefore improves generation quality while avoiding the separate SigLIP2 encoding pass required by direct concatenation.

RepresentationChannelsVFM Required for EncodingEncode TFLOPs/clip↓FVD@50K↓FVD@100K↓
Native VAE baseline: Wan 48ch48✗3.97993.0331.0
Capacity baseline: Wan 96ch, w/o alignment96✗4.34770.0344.7
Raw concatenation baseline: concat-12001200✓7.712855.01896.0
Compressed concatenation baseline: com9696✓7.71740.8294.2
Semantic Mapper: WeaveAE (ours)96✗4.34472.3214.2

All rows use Latte-XL. Encoding costs include the VAE encoder and additional feature extraction.

💡3. WeaveDiT improves generation with each representation held fixed: We compared WeaveDiT with single-stream Latte-XL on three 96-channel representations. WeaveDiT achieves lower FVD at every evaluated checkpoint for all three.

  • (i) The benefit extends beyond semantic–detail separation. **The improvement also holds for the reconstruction-only Wan 96ch latent, where WeaveDiT reduces FVD from 770.0 to 538.9 at 50K steps. This indicates that the architectural benefit is not restricted to latents with explicitly differentiated semantic and detail strands.

  • (ii) The best results come from combining WeaveDiT with WeaveAE. At 100K steps, WeaveDiT reduces FVD from 294.2 to 227.8 for com96 and from 214.2 to 192.1 for WeaveAE. WeaveAE with WeaveDiT achieves the lowest FVD at every checkpoint, supporting the combined use of latent reorganization and dual-strand modeling.

Each entry reports FVD for Latte-XL / WeaveDiT.

RepresentationStrand ConfigurationFVD@50K↓FVD@75K↓FVD@100K↓
Capacity baseline: Wan 96ch, w/o alignmentReconstruction-only770.0 / 538.9431.4 / 362.7344.7 / 301.1
Compressed concatenation baseline: com96Direct semantic + detail concatenation740.8 / 387.2387.3 / 286.3294.2 / 227.8
Semantic Mapper: WeaveAE (ours)Reorganized semantic + detail472.3 / 314.0261.5 / 217.1214.2 / 192.1

📝[W3] Evaluation on Larger Data, Higher Resolution, and Longer Clips

To evaluate beyond the original UCF-101 setting, we conduct additional experiments on Kinetics-700 at 384×384 resolution with 33-frame clips. Relative to the original 256×256, 17-frame configuration, this increases the latent token count by approximately 4×. Both generators are trained on the same data and input configuration for 100K steps.

💡1. Decoder-only adaptation supports the extended configuration: Starting from WeaveAE trained at 256×256 with 17-frame clips, we keep the pretrained VAE encoder, Semantic Mapper, and Projector frozen and adapt only the decoder for 500 steps at 384×384 with 33-frame clips. This improves PSNR from 30.66 to 33.87 and reduces reconstruction FVD from 21.56 to 10.86.

The adapted decoder takes 0.066 seconds per clip, compared with 0.764 seconds for the original Wan2.2 decoder, corresponding to approximately 11.6× faster decoding. Wan2.2 retains better absolute reconstruction metrics, while WeaveAE offers substantially lower decoding latency.

TokenizerPSNR↑SSIM↑LPIPS↓rFVD33↓Decode time / clip
Wan2.2 VAE38.160.9630.0445.070.764 s
WeaveAE, zero-shot30.660.9330.07921.560.066 s
WeaveAE, 500-step decoder adaptation33.870.9480.06810.860.066 s

All reconstruction results above are evaluated on Kinetics-700 at 384×384 resolution with 33-frame clips.

💡2. Generative improvements persist in the extended setting. Using the adapted decoder, LatentWeave achieves lower FVD than Wan2.2 VAE + Latte-XL at every evaluated checkpoint. At 40K steps, LatentWeave reaches an FVD of 682.8, already improving on the baseline's 694.5 at 100K steps. At the full 100K-step budget, LatentWeave achieves 324.4, a 53.3% reduction relative to the baseline.

MethodFVD33@20K↓FVD33@40K↓FVD33@60K↓FVD33@80K↓FVD33@100K↓
Wan2.2 VAE + Latte-XL4793.12514.11072.8783.0694.5
LatentWeave (ours)2851.6682.8523.1432.6324.4

These results support the method’s applicability beyond the original UCF-101 setting, with consistent generation gains on Kinetics-700 at higher resolution and longer clip length.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.