acceptodds
Under review as a conference paper at ICLR 2027

VOCABSCOPE: READING CONCEPTS FROM VIDEO GENERATION STATES IN SPACE AND TIME

Abstract

Video generation models can produce rich and dynamic videos from natural-language prompts, yet their hidden states remain difficult to interpret in terms of concepts. We introduce VocabScope, a framework that maps hidden states of video generators into vocabulary space while preserving their correspondence to video locations. Our method first establishes token-level correspondence between the video generator and a frozen vision–language model, then aligns their representation spaces, and finally connects intermediate representations in its language backbone to vocabulary predictions through layer-specific Jacobian transport. This makes it possible to examine what concept evidence is readable from a generation state and associate it with locations in video space and time. VocabScope offers a direct vocabulary-space view of video-generation hidden states and provides a tool for inspecting concept evidence at corresponding video locations.

Primary area:
interpretability and explainable AI
Keywords:
Video generation interpretability, Vocabulary-space concept

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KvFaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the interpretation of hidden states in a video generation model across decoder layers and denoising steps. The paper introduces a shared alignment layer between the hidden states from the generator and the counter-part from an MLLM, which is trained using typical alignment loss and distribution distillation loss, while both the generator and the MLLM remain frozen. It further implements J-Lens to transfer intermediate representations to the final layer to calculate the similarity score. The method is implemented on AnyFlow with Wan2.1-T2V-1.3B and tested on a self-constructed dataset with 42 test videos.

Critical strengths
  1. The proposed method is simple and easy to implement.

  2. The method shows significant improvement over vanilla baselines such as CLIP.

  3. The problem studied in this paper is interesting. The interpretation of hidden states in a DiT is, in my view, very important, but the study of interpretation should serve the goal to improve generation quality.

Critical weaknesses
  1. The technical contributions of this work appear limited. The embedding alignment loss Lemb\mathcal{L}_{emb} is adopted from REPA, while the position-indexed vocabulary readout is adopted from J-Lens, as acknowledged in the main paper. More fundamentally, the proposed method appears to function more as a distillation approach than an interpretability method. Hidden states at the same spatial positions across different denoising steps may encode distinct semantic information. Distilling these representations indiscriminately into the final hidden states may therefore be inappropriate and could discard meaningful step-specific information.

  2. The topic is interesting, but its applicability and practical significance remain unclear. The paper does not demonstrate or discuss how interpreting intermediate states can benefit video generation or other applications. It also provides little insight into the interpretability of DiT hidden states across decoder blocks and denoising steps. Furthermore, the analysis appears to be conducted primarily in the image domain rather than the video domain. Since the method is not specific to video generation, evaluating it on image generation first may provide a more direct assessment of its effectiveness.

  3. The method uses a universal feature alignment layer across denoising steps and generator blocks, despite the potentially substantial distributional differences between their hidden states. The authors should provide a convincing justification for this design or validate it experimentally against block-specific or step-specific alternatives.

  4. Several design choices appear intuitive and require further ablation or analysis. For example, how were blocks 5, 10, 20, and 28 selected? In addition, the training procedure introduces the distribution distillation loss only after the alignment loss, but the rationale for this training order is unclear. Key hyperparameters, such as λKD\lambda_{KD} and τ\tau, also warrant further investigation.

  5. The main experiments are limited in scale and model coverage. The method is evaluated only on Wan2.1-T2V-1.3B, without testing larger models such as Wan2.1-T2V-14B or newer versions such as Wan2.2. The test set also contains only 42 videos. Moreover, the base Wan2.1-T2V-1.3B model typically uses substantially more denoising steps (around 50). Why do the authors evaluate their method only on the distilled AnyFlow version rather than directly on the base model?

  6. The description of the evaluation metrics is somewhat confusing. At which decoder layers and denoising steps are the metrics computed? Are the results averaged across all decoder layers and denoising steps, or are they computed only at the decoder blocks (5, 10, 20, and 28) used during training? The authors should clarify this evaluation protocol.

Questions for authors

Please see weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KvFa

W1 (contribution, and whether VocabScope is distillation). VocabScope uses a distillation loss, but it is not a distillation method. It is an interpretability method that reads a frozen generator. The generator and the VLM stay frozen, and nothing is distilled into the generator. The distillation term in Equation 5 only trains one affine layer, which serves as the reader. This term makes the VLM's next-token predictions from aligned generator features match those from native features of the same completed training video (Section 3.3). No generator state is used as a target.

At inference, each block–step state is mapped on its own. The readout at a step takes only that state as generator input. At block 10 and step 1, local support is 55.4% with the generator features, 23.1% after spatial shuffling, 20.0% with features from another video and 1.5% with zero features (Table 1).

REPA aligns generator features with a pretrained encoder in order to train the generator. J-Lens reads a language model through its own vocabulary head. A video generator has no vocabulary head. VocabScope matches each position of the aligned generator grid to a slot in the VLM's video-token layout (Section 3.2). One affine layer places the generator feature in that slot (Section 3.3). The VLM's language backbone and vocabulary head then read every position (Section 3.4).

W2 (practical value and insight across blocks and steps). Appendix B uses VocabScope scores from a separate configuration to choose where to edit the generator. The update at the highest-scoring locations for fire, truck and vehicle removes the fire truck. The same update at shifted or random locations leaves it in place (Figure B.1).

Table A.2 gives local support@5 (%) for each block and step.

BlockStep 0Step 1Step 2Step 3Mean
528.550.847.738.541.3
1040.054.649.246.247.5
2046.946.944.640.844.8
2840.839.233.839.238.3

At step 0, local support is already 28.5–46.9% across the four blocks. Table A.1 uses four seeds for 10 test prompts. At every block, the step-0 and step-1 readouts share more words on average with the step-3 readout of the same video than with that of another seed of the same prompt. This holds for all 10 prompts (exact sign-flip test, p = 0.002 for each block and step).

W3 (one alignment layer for all blocks and steps). Designs for each block, each step or each block–step pair need 4, 4 or 16 alignment layers, each with its own Jacobian matrix. VocabScope needs one of each, so all readouts in Tables 3 and A.2 come from the same reader. Table 4 tests three training coverages at the same 16 block–step pairs. B and C use the same training budget.

Training coverageLocal / video support (%)
A: block 10, step 127.8 / 32.5
B: block 10, steps 0–332.4 / 39.6
C: blocks 5/10/20/28, steps 0–343.0 / 50.6

W4 (choice of blocks, training order, λ_KD and τ). Blocks 5, 10, 20 and 28 cover the early, middle and late parts of the 30-block generator. They were fixed when the generator states of the dataset were cached. The cosine loss aligns feature directions, but it does not by itself ensure that the VLM's next-token predictions match those from native features. Distillation therefore starts from the aligned layer and refines it (Section 3.3). With the same initialization, positions and number of updates, local support is 10.7% for distillation only, 38.6% for feature alignment only (λ_KD = 0) and 43.0% for the full objective (λ_KD = 0.1, Table 5). We use τ = 1 (Section 4.1).

W5 (test set and the distilled generator). AnyFlow–Wan2.1-T2V-1.3B is distilled from Wan2.1-T2V-1.3B by flow-map distillation. It has the same 30 blocks and 1,536-dimensional residuals, and supports few-step generation. With four steps, VocabScope is trained and evaluated at every denoising step (Tables 3 and A.2). Tables 1–6 use 26 test videos. Their prompt families do not appear in training. Table A.1 adds three seeds for 10 test prompts.

W6 (evaluation protocol). Local support@5 is the share of the top-5 words at a position that are supported by the matching region and its context in the completed video. Video support@5 also counts words supported elsewhere in the video (Section 4.1). In Tables 1–6, A.1 and A.2, VocabScope reads generator features only at blocks 5, 10, 20 and 28. These are the training blocks of configuration C. It reads words at language-backbone layer 26 with J26J_{26} in Tables 1–5. In Tables 1 and 2, it reads block 10 at step 1 (130 judgments per row). Table 3 averages the four blocks at each step (520 per step). Tables 4–6 average all 16 block–step pairs with equal weight (2,080 per setting). Table A.2 reports each pair (130 per cell).

I thank the authors for their response. However, I believe several of my main concerns remain insufficiently addressed:

  1. (W1) The core technical components largely overlap with existing works, making the overall contribution rather limited.
  2. (W3 & W4) The sensitivity to key hyperparameters has not been adequately investigated.
  3. (W5) It remains unclear whether the proposed method generalizes beyond Wan2.1-1.3B. Furthermore, the small test set raises additional concerns about the reliability and representativeness of the evaluation results.

Overall, I will maintain my original rating.

Reviewer_RjFPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VocabScope, a method for reading the intermediate hidden states of a video generator as words at specific video locations. Generator features are aligned to the video-token grid of a frozen VLM, mapped into its input space with a single trained affine layer, and read out through the VLM's vocabulary, using layer-wise Jacobians to read intermediate layers. Experiments on Wan2.1-based AnyFlow with 280 generated videos evaluate whether the top-5 words match the corresponding region, with spatial-shuffle and zero-feature controls, comparisons with CLIP and IMAP, and ablations of training coverage, losses, and Jacobian transport.

Critical strengths
  1. The proposed analysis method is simple and general. It reads video-generator hidden states directly in the vocabulary of a pretrained VLM while keeping each reading tied to a video location, without specifying a concept set in advance. Only one affine layer is trained and both models stay frozen, so the method is cheap to fit and easy to apply to new generation runs. This is consequential because it makes an open-vocabulary, location-indexed view of generator states practical.
  2. The control experiments provide evidence that the readout is position-specific. Spatially shuffling generator features drops local support from 55.4% to 23.1% while video support stays at 50.8%, and zero features give 1.5%. This is consequential because it shows that the words come from the features at that location, rather than from global video content or the VLM's instruction alone.
Critical weaknesses
  1. The paper shows that generator features can be mapped to plausible words at the right locations, but it does not demonstrate that VocabScope is a useful tool for analyzing video generators. A single feature could support many different top-5 words, and the experiments mostly show static object and scene words. There is no analysis of motion or action concepts, of how concepts evolve across frames (only 3 of 21 latent frames are used despite the title's "space and time"), or of whether the readout helps diagnose failed generations. This is consequential because the paper's value as an interpretability tool rests on such evidence.

  2. The evaluation is not reliable enough to support the quantitative claims. Missing labels in Tables 2, 5 and 6 were "supplemented with AI-assisted judgments pending human verification", with no details on the model or the fraction affected. The number of evaluated positions and the annotation protocol are not reported, and no annotations or evaluation scripts are provided. The baselines are weak (CLIP on the noisy latent scores 0.0%), and at the layer used for the main results, Jacobian transport changes local support only from 42.3% to 43.0% (Table 6). As a result, the comparisons in Tables 2–6 cannot be trusted as reported.

Questions for authors
  1. The evaluation uses "matched positions" in held-out videos. If this means that a subset of positions was selected rather than all positions, how many positions were evaluated in each table, and how were they selected? Were the annotators blind to the method?
  2. Which AI model and prompt produced the AI-assisted labels, what fraction of labels in Tables 2, 5 and 6 did it fill, and do the numbers change after human verification?
  3. Which AI model and prompt produced the AI-assisted labels, what fraction of labels in Tables 2, 5 and 6 did it fill, and do the numbers change after human verification?
  4. For one prompt with several seeds, do step-0 and step-1 readouts follow seed-specific layout, or are they largely prompt-determined?
  5. How does VocabScope compare with (a) CLIP applied to the decoded predicted clean sample at each step, rather than the noisy latent, and (b) the same readout applied to the VLM's native features of the final video, as an upper bound?
  6. Can VocabScope read motion or action concepts and track how they change across frames, or help diagnose failed or temporally inconsistent generations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer RjFP

W1/Q6 (VocabScope as an analysis tool). VocabScope scores the full vocabulary of the frozen VLM at each aligned video position. It does this for a chosen generator block and denoising step (Section 3.1). The paper uses these scores in three ways.

First, local support depends on the generator feature that is read (Table 1).

ConditionLocal / video support (%)
Generator features55.4 / 57.7
Spatially shuffled features23.1 / 50.8
Features from another video20.0 / 34.6
No attention between video tokens50.0 / 56.2

In the last row, attention between video tokens is blocked. Each readout then uses only the feature at its own position and the fixed text before the video.

Second, the readout shows when words that match the completed video appear. Averaged over blocks 5, 10, 20 and 28, local support is 39.0% at step 0, 47.9% at step 1, 43.8% at step 2 and 41.2% at step 3 (Table 3). Table A.2 gives these numbers for each block.

Third, the scores choose where to edit the generator. Appendix B uses a separate configuration. We apply the same feature update at three kinds of locations. At the locations with the highest scores for fire, truck and vehicle, the update removes the fire truck. At shifted or random locations, the truck stays (Figure B.1).

W2 (reliability of the evaluation). Section 4.1 gives the positions and judgments for Tables 1–6. All labels are verified by human annotation (Appendix A). Unsupported and uncertain words count in the denominator.

All methods in Table 2 rank the same 2,245 candidates. Two of them read the generation at step 1, as VocabScope does. IMAP (GramCol) uses the generator's attention features. Decoded patch + CLIP applies CLIP to the VAE reconstruction of the noisy latent. VocabScope reaches 54.6% local support, IMAP reaches 22.3% and Decoded patch + CLIP reaches 0.0%. Table 2 also reports Final patch + CLIP, which reads the completed video.

Jacobian transport lets the frozen vocabulary head read earlier layers (Section 3.4). At layers 1, 9 and 17, it raises local support from 5.9%, 11.8% and 18.4% to 32.8%, 34.8% and 38.3%. At layer 26, the results with and without it are close (43.0% and 42.3%, Table 6).

Q1 (which positions are evaluated). Yes, it is a subset. All rows of Tables 1–6 use the same 26 positions, one in each of 26 test videos, at temporal position 2. In Tables 1 and 2, VocabScope reads block 10 at step 1 (130 judgments per row). In Table 3, it reads blocks 5, 10, 20 and 28 at each step (520 per step). Tables 4–6 use all 16 block–step pairs (2,080 per setting). Appendix A describes how the positions were selected and annotated.

Q2/Q3 (AI model, prompt and share of AI drafts). The drafts were made by the OpenAI Codex agent. For each position it drafted, it saw the outputs grouped by method, the 32×32 cell in frame 8, its 96×96 context and nine frames of the completed video. It did not see the generation prompt or any aggregate score. It marked words as local, elsewhere or uncertain. All other words were labeled unsupported. Every draft was verified by human annotation, and wrong labels were corrected. Labels that started as AI drafts make up 11–43% of each row in Table 1. In Table 2, they make up 0%, 2%, 89% and 34% of the Decoded patch + CLIP, Final patch + CLIP, IMAP and VocabScope rows. They make up 53–71% per step in Table 3, and 37–63%, 61–90% and 61–91% per setting in Tables 4, 5 and 6 (Appendix A). All reported numbers in Tables 1–6 come from these verified labels. Table 2 uses configuration C.

Q4 (seed dependence of early readouts). We chose 10 test prompts at random and generated four seeds for each. We read all 390 positions of temporal position 2 at blocks 5, 10, 20 and 28 (Appendix A, Table A.1). We compare each early readout with the step-3 readout of the same video (own) and of another seed of the same prompt (other). At block 10, the step-0 readout shares on average 0.89 of its five words with own and 0.68 with other. At step 1, these averages are 1.84 and 0.86. Own is higher for all 10 prompts at every block and both steps (exact sign-flip test over prompts, p = 0.002 for each block and step). So early readouts agree more with the step-3 readout of their own video than with the step-3 readout of another seed.

Q5 (CLIP on the predicted clean sample and native VLM features). Both comparisons need pixels. (a) needs a VAE decode at every step. At steps 0–2, it also needs an extra generator forward pass to predict the clean sample. (b) needs the completed video. VocabScope reads the hidden state directly, with no extra pass and no decoding. (a) gives one readout per step. VocabScope gives one readout for each block at each step. The native VLM features of completed training videos are the targets of both training objectives (Section 3.3).

Thank you for completing the human verification of the labels (Q2/Q3), clarifying the position counts (Q1), and adding the seed-dependence (Q4) and attention-blocking (Table 1) experiments. However, my core concerns remain unresolved: all quantitative tables evaluate only 26 positions at a single fixed time index (temporal position 2) without showing motion/action concepts, cross-frame evolution, or failure diagnosis (W1/Q6), and the response omits the requested predicted clean-sample CLIP and native VLM feature upper-bound comparisons (W2/Q5) while Jacobian transport remains marginal at the main readout layer (43.0% vs. 42.3%). Given the small single-time evaluation scale and the lack of video-specific insights or reference comparisons, I will maintain my score of 2 (Weak Rejection).

Reviewer_Fy5ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VocabScope, a probe for reading location-indexed vocabulary scores from intermediate states of a video generator. It temporally and spatially aligns pooled generator features with a frozen Qwen3-VL visual-token layout, learns one shared affine map using completed-video visual features and description-distribution distillation, and transports language-backbone activations to the final vocabulary head with layer-specific averaged Jacobians. On 42 held-out videos from prompt-disjoint families, it reports state-dependence controls, comparisons to two CLIP baselines and IMAP, coverage/objective/Jacobian ablations, and one concept-guided editing illustration.

Critical strengths
  1. The paper identifies a useful missing interface: existing diffusion/video analyses localize prescribed concepts or discover latent features, whereas this work attempts to expose arbitrary vocabulary evidence at a video location.

  2. The paper is clearly written and easy to follow. The method figure and qualitative examples make the proposed workflow and its outputs easy to understand.

Critical weaknesses
  1. The technical novelty needs clearer support. Prior work has used learned visual–language projections [1] and output distillation [2], while ConceptAttention [3] localizes supplied concepts and J-Lens [4] provides Jacobian-based vocabulary readout. The contribution combines these ideas for position-indexed video-state analysis; its advantage over simpler alternatives remains unclear.

  2. The baseline comparison is insufficient (Table 2), making the benefit of the complete readout difficult to judge. Comparisons with a trained feature-to-vocabulary probe and other relevant internal-feature methods are needed to assess the contribution of the proposed readout.

[1] Liu et al. Visual Instruction Tuning. NeurIPS 2023.

[2] Ji and Zhu. Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher. NeurIPS 2020.

[3] Helbling et al. ConceptAttention: Diffusion Transformers Learn Highly Interpretable Features. ICML 2025.

[4] Gurnee et al. Verbalizable Representations Form a Global Workspace in Language Models. Transformer Circuits technical report, 2026.

Questions for authors
  1. What does the proposed combination add beyond existing alignment and vocabulary-readout techniques, and how does it differ from prior concept-localization methods?

  2. Can the authors distinguish concepts represented at a particular generation step from information that predicts the completed video, or clarify the intended scope of the readout?

  3. Can the authors further test how much each prediction depends on its queried position, for example by limiting cross-position information or comparing local and global inputs?

  4. Can the authors compare against a simple trained feature-to-vocabulary probe or another method operating on generator features under matched data and evaluation conditions?

  5. How many positions and judgments support each main table, and what are the results after completing human verification of the AI-assisted labels?

  6. Does the method remain effective with an external candidate vocabulary or a different generator–VLM pair, and can the intervention be tested on additional concepts or videos?

2 comments

Response to Reviewer Fy5Z

W1/Q1 (novelty and difference from concept localization). VocabScope asks which concepts a video generator's hidden state carries for each video location, and answers with scores over the full vocabulary of a frozen VLM (Table 1), with no concept list as input and no concept labels in training. Concept-localization methods (ConceptAttention, IMAP) ask the reverse question, taking the concept as input and computing one map per supplied concept. J-Lens needs the inspected model to have its own vocabulary head, which a video generator lacks. LLaVA-style projectors take image-encoder features of a finished image as input. At readout time, VocabScope needs none of these. Each pooled generator feature is placed in the matching slot of the VLM's video-token layout (Section 3.2), one affine layer maps it into the VLM's visual input space (Section 3.3), and Jacobian transport carries the VLM's intermediate representation at that slot to the vocabulary head (Section 3.4). Every aligned position therefore gets its own vocabulary readout.

Tables 5 and 6 isolate the objective and the Jacobian transport, which makes earlier VLM layers readable.

SettingLocal support@5 (%)
Distillation only10.7
Feature alignment only (Eq. 4)38.6
Full objective (Eq. 5)43.0
Layer 1, without / with Jacobian transport5.9 / 32.8
Layer 9, without / with Jacobian transport11.8 / 34.8

W2/Q4 (baselines on generator features). Table 2 includes IMAP (GramCol), adapted to rank the same 2,245 candidates from the generator's attention features at the same 26 positions and denoising step (Section 4.1, Appendix A). Methods that read the intermediate state at step 1 give:

Method at step 1InputLocal / video support@5 (%)
Decoded patch + CLIPVAE reconstruction of the noisy latent0.0 / 0.0
IMAP (GramCol)Generator attention features, blocks 5–1922.3 / 36.2
VocabScopeGenerator hidden state, block 1054.6 / 59.2

Table 2 also reports Final patch + CLIP, which reads the completed video. ConceptAttention builds its maps in the joint text–image attention of MM-DiT generators, which Wan2.1 does not have.

Q2 (scope of the readout). The readout's scope is the concept evidence that the generator state at a given step and block carries for each location, expressed in the VLM vocabulary. The alignment layer is trained with native VLM features and descriptions of completed training videos (Section 3.3). The completed test video is the visual reference for scoring, not ground truth for which concepts an intermediate state represents (Section 4.1). Local and video support therefore measure how much of this evidence agrees with the completed video at each step (Table 3).

Q3 (dependence on the queried position). A new row in Table 1 keeps the generator features and blocks attention between video tokens in every language-backbone layer of the VLM. The readout at each position then depends only on the aligned feature at that position and the fixed template text before the video.

Condition (Table 1, full vocabulary)Local support@5 (%)Video support@5 (%)
Generator features55.457.7
No attention between video tokens50.056.2
Spatially shuffled features23.150.8

Q5 (positions and judgments per table). Section 4.1 lists these counts.

TableEvaluated positionsJudgments
1, 226, one in each of 26 test videos130 per row
3The same 26 at blocks 5, 10, 20, 28520 per step
4, 5, 6The same 26 at blocks 5, 10, 20, 28 and steps 0–32,080 per setting

All labels in Tables 1–6 are verified by human annotation, and the reported numbers are computed from these labels.

Q6 (external vocabulary, other model pairs and intervention). VocabScope requires no candidate list. Table 1 ranks the full Qwen3-VL vocabulary, which contains every single-token word an external list could supply, at generator block 10 and denoising step 1 on the 26 test videos (Sections 4.1–4.2). Over this vocabulary, generator features give 55.4% local support, against 20.0% with features from another video. Tables 2–6 rank a shared list of 2,245 words from training prompts for all methods and settings (Appendix A). For a generator–VLM pair, VocabScope sets up the grid correspondence of Section 3.2 and fits one affine layer and its Jacobian, with both models frozen. The reported pair is AnyFlow–Wan2.1-T2V-1.3B with Qwen3-VL-2B-Instruct (Section 4.1). In Appendix B, VocabScope scores from a separate configuration for fire, truck and vehicle select where to edit the generator. The update at the selected locations removes the fire truck, while the same update at shifted or random masks leaves it in place (Figure B.1).

Thank you for answering my questions. However, some major concerns remain unresolved. For example, the baseline evidence is still insufficient to establish the effectiveness of VocabScope. Table 2 compares against CLIP-based variants and an adapted IMAP baseline, but it does not include stronger or more recent baselines. It also lacks a direct feature-to-vocabulary probe on the same generator states. The reported gain therefore does not yet show how much comes from the full method itself. Therefore, my score remains unchanged.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

VocabScope reads concepts from video-generator hidden states in the vocabulary of a frozen vision–language model (VLM), at each aligned position in space and time. Concept-localization methods such as ConceptAttention and IMAP need the concept as input, and J-Lens needs the inspected model to have its own vocabulary head, which a video generator lacks. VocabScope scores the VLM's full vocabulary at a generator location, with no concept list as input and no concept labels in training. Over the full vocabulary at block 10 and step 1, its top-5 words reach 55.4% local support, against 23.1% after spatial shuffling and 20.0% with features from another video (Table 1). Among methods that read the generation at step 1 and rank the same 2,245 candidates, VocabScope reaches 54.6% local support, against 0.0% for CLIP on the decoded patch and 22.3% for IMAP (Table 2). Table 2 also reports Final patch + CLIP, which reads the completed video. The revised PDF contains every change below.

Evaluation protocol and labels (Fy5Z Q5, RjFP W2, Q1, Q2/Q3, KvFa W6). Section 4.1 gives the number of positions and judgments for Tables 1–6. They use one position in each of 26 test videos from prompt families disjoint from training. VocabScope reads language-backbone layer 26 with J26J_{26}, estimated from layer 26 to the final layer 27, in Tables 1–5.

TableVocabScope readsJudgments
1, 2block 10, step 1130 per row
3blocks 5, 10, 20, 28 at each step520 per step
4–6blocks 5, 10, 20, 28 × steps 0–32,080 per setting

All labels are verified by human annotation, and the reported numbers are computed from the verified labels. Appendix A, now "Evaluation and Baseline Details", describes how the 26 positions were obtained, the annotation and AI-assisted drafting protocol, and the share of labels that started as AI drafts in each table.

Cross-position control (Fy5Z Q3, RjFP W1/Q6). A new row in Table 1 blocks attention between video tokens in every language-backbone layer, so each readout depends only on the aligned feature at that position and the fixed text before the video. It gives 50.0% local support, against 55.4% with unmodified attention.

Baselines (Fy5Z W2/Q4, RjFP W2). Table 2 and Figure 3 use configuration C, and all rows of Table 2 are scored with one label set verified by human annotation. Appendix A details each baseline. IMAP ranks the same 2,245 candidates from the generator's attention features. ConceptAttention builds its maps in the joint text–image attention of MM-DiT generators, which Wan2.1 does not have.

Jacobian transport (Fy5Z W1/Q1, RjFP W2). Transport makes earlier language-backbone layers readable through the frozen vocabulary head. At layers 1, 9 and 17, it raises local support from 5.9%, 11.8% and 18.4% to 32.8%, 34.8% and 38.3% (Table 6).

Seeds, blocks and steps (RjFP Q4, KvFa W2–W5). Table A.1 compares four seeds of 10 test prompts. At blocks 5, 10, 20 and 28, the step-0 and step-1 readouts share more words with the step-3 readout of the same video than with that of another seed of the same prompt, for all 10 prompts (p = 0.002 at each block and step). Table A.2 reports each block and step. With one alignment layer, local support averaged over steps is 38.3–47.5% per block, and averaged over blocks it is 39.0–47.9% per step.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.