- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the interpretation of hidden states in a video generation model across decoder layers and denoising steps. The paper introduces a shared alignment layer between the hidden states from the generator and the counter-part from an MLLM, which is trained using typical alignment loss and distribution distillation loss, while both the generator and the MLLM remain frozen. It further implements J-Lens to transfer intermediate representations to the final layer to calculate the similarity score. The method is implemented on AnyFlow with Wan2.1-T2V-1.3B and tested on a self-constructed dataset with 42 test videos.
Critical strengths
-
The proposed method is simple and easy to implement.
-
The method shows significant improvement over vanilla baselines such as CLIP.
-
The problem studied in this paper is interesting. The interpretation of hidden states in a DiT is, in my view, very important, but the study of interpretation should serve the goal to improve generation quality.
Critical weaknesses
-
The technical contributions of this work appear limited. The embedding alignment loss is adopted from REPA, while the position-indexed vocabulary readout is adopted from J-Lens, as acknowledged in the main paper. More fundamentally, the proposed method appears to function more as a distillation approach than an interpretability method. Hidden states at the same spatial positions across different denoising steps may encode distinct semantic information. Distilling these representations indiscriminately into the final hidden states may therefore be inappropriate and could discard meaningful step-specific information.
-
The topic is interesting, but its applicability and practical significance remain unclear. The paper does not demonstrate or discuss how interpreting intermediate states can benefit video generation or other applications. It also provides little insight into the interpretability of DiT hidden states across decoder blocks and denoising steps. Furthermore, the analysis appears to be conducted primarily in the image domain rather than the video domain. Since the method is not specific to video generation, evaluating it on image generation first may provide a more direct assessment of its effectiveness.
-
The method uses a universal feature alignment layer across denoising steps and generator blocks, despite the potentially substantial distributional differences between their hidden states. The authors should provide a convincing justification for this design or validate it experimentally against block-specific or step-specific alternatives.
-
Several design choices appear intuitive and require further ablation or analysis. For example, how were blocks 5, 10, 20, and 28 selected? In addition, the training procedure introduces the distribution distillation loss only after the alignment loss, but the rationale for this training order is unclear. Key hyperparameters, such as and , also warrant further investigation.
-
The main experiments are limited in scale and model coverage. The method is evaluated only on Wan2.1-T2V-1.3B, without testing larger models such as Wan2.1-T2V-14B or newer versions such as Wan2.2. The test set also contains only 42 videos. Moreover, the base Wan2.1-T2V-1.3B model typically uses substantially more denoising steps (around 50). Why do the authors evaluate their method only on the distilled AnyFlow version rather than directly on the base model?
-
The description of the evaluation metrics is somewhat confusing. At which decoder layers and denoising steps are the metrics computed? Are the results averaged across all decoder layers and denoising steps, or are they computed only at the decoder blocks (5, 10, 20, and 28) used during training? The authors should clarify this evaluation protocol.
Questions for authors
Please see weaknesses.
2 comments
I thank the authors for their response. However, I believe several of my main concerns remain insufficiently addressed:
- (W1) The core technical components largely overlap with existing works, making the overall contribution rather limited.
- (W3 & W4) The sensitivity to key hyperparameters has not been adequately investigated.
- (W5) It remains unclear whether the proposed method generalizes beyond Wan2.1-1.3B. Furthermore, the small test set raises additional concerns about the reliability and representativeness of the evaluation results.
Overall, I will maintain my original rating.
Response to Reviewer KvFa
W1 (contribution, and whether VocabScope is distillation). VocabScope uses a distillation loss, but it is not a distillation method. It is an interpretability method that reads a frozen generator. The generator and the VLM stay frozen, and nothing is distilled into the generator. The distillation term in Equation 5 only trains one affine layer, which serves as the reader. This term makes the VLM's next-token predictions from aligned generator features match those from native features of the same completed training video (Section 3.3). No generator state is used as a target.
At inference, each block–step state is mapped on its own. The readout at a step takes only that state as generator input. At block 10 and step 1, local support is 55.4% with the generator features, 23.1% after spatial shuffling, 20.0% with features from another video and 1.5% with zero features (Table 1).
REPA aligns generator features with a pretrained encoder in order to train the generator. J-Lens reads a language model through its own vocabulary head. A video generator has no vocabulary head. VocabScope matches each position of the aligned generator grid to a slot in the VLM's video-token layout (Section 3.2). One affine layer places the generator feature in that slot (Section 3.3). The VLM's language backbone and vocabulary head then read every position (Section 3.4).
W2 (practical value and insight across blocks and steps). Appendix B uses VocabScope scores from a separate configuration to choose where to edit the generator. The update at the highest-scoring locations for fire, truck and vehicle removes the fire truck. The same update at shifted or random locations leaves it in place (Figure B.1).
Table A.2 gives local support@5 (%) for each block and step.
At step 0, local support is already 28.5–46.9% across the four blocks. Table A.1 uses four seeds for 10 test prompts. At every block, the step-0 and step-1 readouts share more words on average with the step-3 readout of the same video than with that of another seed of the same prompt. This holds for all 10 prompts (exact sign-flip test, p = 0.002 for each block and step).
W3 (one alignment layer for all blocks and steps). Designs for each block, each step or each block–step pair need 4, 4 or 16 alignment layers, each with its own Jacobian matrix. VocabScope needs one of each, so all readouts in Tables 3 and A.2 come from the same reader. Table 4 tests three training coverages at the same 16 block–step pairs. B and C use the same training budget.
W4 (choice of blocks, training order, λ_KD and τ). Blocks 5, 10, 20 and 28 cover the early, middle and late parts of the 30-block generator. They were fixed when the generator states of the dataset were cached. The cosine loss aligns feature directions, but it does not by itself ensure that the VLM's next-token predictions match those from native features. Distillation therefore starts from the aligned layer and refines it (Section 3.3). With the same initialization, positions and number of updates, local support is 10.7% for distillation only, 38.6% for feature alignment only (λ_KD = 0) and 43.0% for the full objective (λ_KD = 0.1, Table 5). We use τ = 1 (Section 4.1).
W5 (test set and the distilled generator). AnyFlow–Wan2.1-T2V-1.3B is distilled from Wan2.1-T2V-1.3B by flow-map distillation. It has the same 30 blocks and 1,536-dimensional residuals, and supports few-step generation. With four steps, VocabScope is trained and evaluated at every denoising step (Tables 3 and A.2). Tables 1–6 use 26 test videos. Their prompt families do not appear in training. Table A.1 adds three seeds for 10 test prompts.
W6 (evaluation protocol). Local support@5 is the share of the top-5 words at a position that are supported by the matching region and its context in the completed video. Video support@5 also counts words supported elsewhere in the video (Section 4.1). In Tables 1–6, A.1 and A.2, VocabScope reads generator features only at blocks 5, 10, 20 and 28. These are the training blocks of configuration C. It reads words at language-backbone layer 26 with J26 in Tables 1–5. In Tables 1 and 2, it reads block 10 at step 1 (130 judgments per row). Table 3 averages the four blocks at each step (520 per step). Tables 4–6 average all 16 block–step pairs with equal weight (2,080 per setting). Table A.2 reports each pair (130 per cell).