- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed VidMem, which extends the compression-based capacity measurement of Morris et al. (2026) from text to video keys. Generated 16-frame 64×64 clips, or text-token / keyless keys, are bound to independent uniformly random answer strings (64 tokens over a vocabulary of 2048, i.e. 704 bits). Stored information is measured as the answer entropy minus the model's conditional codelength, truncated at zero and maximized over a grid of dataset sizes.
Critical strengths
As mentioned in paper, keys are generated deterministically on a lattice with guaranteed separation, so key crowding cannot fake a plateau, and held-out-index checks show 0 bits on unseen indices.
Besides, a pinned measurement pool, 16 multi-seed points with ≤1% spread, and consistently conservative (lower-bound) are reported to show its robustness.
Critical weaknesses
- The answers are random text strings, and the video serves only as an index. What VidMem measures is the capacity of a small from-scratch decoder to store random key to text associations, not the memorization of video content. The paper acknowledges that "through the answer channel rather than pixel storage" is not very proper.
2.Although both the title, abstract, and introduction motivate the work by memorization in "video-language models" trained on web-scale corpora, by knowledge displacement, and by exposure of sample-specific visual information. None of the measured models is a pretrained VLM. The practically relevant setting, e.g. LoRA or full fine-tuning of a pretrained video VLM, and any visual-content storage, is not studied.
- Eq. 5 is an accounting identity once a valid ceiling C* is assumed. Since C* is not available, the empirical test uses the union-training ceiling estimated from the same runs (a consistency check, as the authors note), while the independent grid estimate mispredicts onset by up to two chunks.
Questions for authors
Can you apply the protocol to a pretrained video VLM (e.g. Qwen2.5-VL-3B), measuring the bits stored by LoRA or full fine-tuning on video-keyed random answers? This is the setting the introduction motivates (knowledge displacement during continued training).
Please report an affine fit C = αP − β (or the fit with the stem excluded) alongside the origin-through fit. Does a single α then describe the video and text tracks across sizes, and how large is β relative to stem size?
Can the ledger be tested against a ceiling estimated independently of the stream being evaluated, e.g. union training on a different stream of the same total entropy? Otherwise the onset agreement is circular.
5 comments
Response to Reviewer bkDp: Weakness 2 and Question 1
Weakness 2. Although both the title, abstract, and introduction motivate the work by memorization in "video-language models" trained on web-scale corpora, by knowledge displacement, and by exposure of sample-specific visual information. None of the measured models is a pretrained VLM. The practically relevant setting, e.g. LoRA or full fine-tuning of a pretrained video VLM, and any visual-content storage, is not studied.
Question 1. Can you apply the protocol to a pretrained video VLM (e.g. Qwen2.5-VL-3B), measuring the bits stored by LoRA or full fine-tuning on video-keyed random answers? This is the setting the introduction motivates (knowledge displacement during continued training).
Answer. Thank you for proposing this experiment, which we ran on the model you suggested with the vision tower frozen, 16-frame appearance keys and 64-token random answers. The protocol applies to this pretrained model. Attention adapters store about one bit per adapter parameter when answers are read over the alphabet used in training, the video selects the stored answer in every tested configuration, and every tested fine-tune raises the codelength of the base model's own captions on held-out clips, by an amount that follows the optimization setting rather than the amount stored.
All three results below are new. They measure storage, key dependence and displacement on the same fine-tuned weights.
- A new capacity measurement shows that attention adapters store about one bit per adapter parameter under the training alphabet. Answers can be coded over the 2,048-symbol alphabet used in training or under the model's full output vocabulary. Under the training alphabet, LoRA adapters on the attention projections reach 1.11 to 1.33 bits per adapter parameter at their best dataset size, depending on the rank, and these values are lower bounds. Under the full vocabulary they store far less, because training with the restricted softmax never penalized probability outside the alphabet; in an additional run, a rank-4 adapter trained with the full-vocabulary loss stores 1.55 Mbit under the full vocabulary, 0.84 bits per adapter parameter against 0.01 with the restricted loss. Under the training alphabet, full-rank fine-tunes of the attention projections or of the whole language model store at least 98.9% of the dataset entropy at every tested dataset size, so they are data-limited and give lower bounds rather than capacities. Appendix Table 22 and Appendix F.
- A new key test shows that the video selects the stored answer in every tested configuration. We gave every sample another sample's video on the same fine-tuned weights. At the first answer token, which has no prefix, the wrong video raises the codelength of the true token above the 11 bits of a uniform guess in every run that stored more than 5% of the dataset entropy. When only attention is adapted, by LoRA at any rank or by full-rank fine-tuning, these runs then lose almost all stored bits. When the MLP blocks are trained as well, by a low-rank adapter or by fine-tuning the whole language model, the true first token costs 27 to 38 bits under a wrong video, and greedy decoding without an answer prefix follows the video shown. Under teacher forcing, the rest of an answer costs little once its first tokens are given, so a wrong video costs these runs only 41 to 66 of the 704 bits per sample. Appendix Table 23.
- A new displacement probe shows that displacement follows the optimization setting rather than the amount stored. For every fine-tuned model of Appendix Table 22 we measured the change in the base model's codelength of its own captions on held-out natural clips. Every fine-tune raised it, including one that stored almost nothing. At equal stored bits, a lower learning rate reduced the increase by an order of magnitude for attention fine-tuning and about sixfold for the whole language model, and a fine-tune on natural captions displaced the model's own captions as much as the random-code fine-tune with the same adapted modules, dataset size, learning rate and number of training steps. Appendix Tables 24 and 25.
Revisions to the manuscript. A new pretrained-model paragraph in Section 4.2 reports results 1 to 3, and Appendix Figure 10 shows captions of held-out clips after fine-tuning. Appendix F gives the runs of Appendix Tables 22 to 25 in full.
Response to Reviewer bkDp: Weakness 3 and Question 3
Weakness 3. Eq. 5 is an accounting identity once a valid ceiling C* is assumed. Since C* is not available, the empirical test uses the union-training ceiling estimated from the same runs (a consistency check, as the authors note), while the independent grid estimate mispredicts onset by up to two chunks.
Question 3. Can the ledger be tested against a ceiling estimated independently of the stream being evaluated, e.g. union training on a different stream of the same total entropy? Otherwise the onset agreement is circular.
Answer. Thank you for proposing this test, which we ran. Eq. 5 is a bound conditional on the unknown ceiling C★, as Section 3.3 states. What we test empirically is an onset rule whose reference is estimated on other streams: a realization rate calibrated on the submitted streams predicted the onset of all ten new streams trained with the same recipe, including four chosen so that it and the grid estimate disagree.
The original submission's test used the realized retention maximum of the same stream, a consistency check. The new test fixes ρ̄ as the mean over the six submitted streams of the realization rate ρ = C_stream/Ĉ, the realized retention maximum C_stream over the grid estimate Ĉ, and predicts the first lossy chunk of each new stream from ρ̄ and the grid estimate as k̂ = ⌊ρ̄Ĉ/H⌋ + 1, where H is the answer entropy of one chunk. No retention measured on the new stream enters the prediction, and its observed onset is read from the retention of each chunk with a fixed threshold of 0.96.
- Six new streams with fresh key and answer seeds show that the calibrated rule predicts every onset. At three model sizes the rule predicts all six onsets, whereas the unscaled grid estimate predicts three. Appendix Table 21.
- Four pre-registered streams show that the rule holds where it and the grid estimate disagree. The rule, the predictions and the onset criterion were recorded before training, and the streams were chosen so that the two rules predict different chunks. Every onset fell at the chunk the rule predicted, and the grid estimate predicted a later onset in every case. Appendix Table 21.
When the grid estimate alone misses, it predicts a later chunk, because union training realizes about 80% of it, and the calibrated rate corrects this.
Revisions to the manuscript. Section 3.3 states the ledger as conditional on C★, keeps Ĉ as a lower bound and renames C_stream the realized retention maximum, which is no longer called a ceiling. The onset paragraph of Section 4.2 reports both tests, and Appendix Table 21 lists every stream.
Response to Reviewer bkDp: Weakness 1
Weakness 1. The answers are random text strings, and the video serves only as an index. What VidMem measures is the capacity of a small from-scratch decoder to store random key to text associations, not the memorization of video content. The paper acknowledges that "through the answer channel rather than pixel storage" is not very proper.
Answer. Thank you for this point on what is stored. VidMem measures how much answer information a video-language model can store and retrieve under a video key, with the video as the key and the answer string as the stored object. This is how such a model uses what it stores: it takes video as input and produces text. Section 3.1 now defines the measured quantity in these terms, and the sentence you quote is removed.
We added a key test that measures how much of the stored answer information depends on the video key, and we applied the protocol to the pretrained model you suggested, Appendix Tables 22 and 23.
A new key test shows that most of the pooled appearance-keyed capacity depends on the correct key. If the model ignored the key during retrieval, giving every sample another sample's key on the same weights would leave compression unchanged; the observed drop therefore demonstrates dependence on the correct key. For appearance keys the key-dependent pooled slope is 2.32 bits per parameter against 2.54 for total compression, with more than 90% of the capacity depending on the key at the three largest tested sizes, which dominate the pooled fit. Text keys depend on the key at every size and motion keys little in the pooled fit, so the test does detect storage that does not need the key where it occurs. Appendix Tables 17 and 18.
Revisions to the manuscript. Section 3.1 defines the measured quantity as the answer compression that a model achieves under a given key configuration, and a new paragraph at the end of Section 3.3 states the scope of each measured quantity. The binding-cost paragraph of Section 4.2 reports the key-dependent reading for each key type.