acceptodds
Under review as a conference paper at ICLR 2027

VidMem: Measuring Memorization Capacity and Knowledge Displacement in Video-Language Models

Abstract

How many bits can a video-language model store in its parameters? This capacity constrains corpus fit, knowledge displacement during continued training, and exposure of sample-specific information, yet it has not been measured for visual modalities. Existing studies detect replication through similarity, extraction, or attack success but lack a common unit and capacity limit. We propose VidMem, which binds procedurally generated videos to independent random answer strings of known entropy and measures stored information as the reduction in conditional codelength from a uniform reference. Across multiple sizes of a calibration decoder family, the estimate scales linearly at 2.54 bits per trained parameter over the tested range. The protocol applies to a second decoder family and five frozen pretrained encoders, although the coefficient is architecture- and format-specific. Pooled slopes indicate that a video-keyed association costs 1.28× as many parameters as a text-keyed one; tested ablations associate this gap with perception parameters and key length. A capacity ledger then characterizes unavoidable forgetting relative to a specified ceiling. Across three chunk-entropy ratios, onset is consistent with the stream-specific capacity realized by union training. VidMem therefore gives multimodal memorization an explicit unit and budget.

Primary area:
interpretability and explainable AI
Keywords:
memorization, model capacity, video-language models, multimodal large language models, conditional compression, information theory, continual learning, catastrophic forgetting

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_bkDpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed VidMem, which extends the compression-based capacity measurement of Morris et al. (2026) from text to video keys. Generated 16-frame 64×64 clips, or text-token / keyless keys, are bound to independent uniformly random answer strings (64 tokens over a vocabulary of 2048, i.e. 704 bits). Stored information is measured as the answer entropy minus the model's conditional codelength, truncated at zero and maximized over a grid of dataset sizes.

Critical strengths

As mentioned in paper, keys are generated deterministically on a lattice with guaranteed separation, so key crowding cannot fake a plateau, and held-out-index checks show 0 bits on unseen indices.

Besides, a pinned measurement pool, 16 multi-seed points with ≤1% spread, and consistently conservative (lower-bound) are reported to show its robustness.

Critical weaknesses
  1. The answers are random text strings, and the video serves only as an index. What VidMem measures is the capacity of a small from-scratch decoder to store random key to text associations, not the memorization of video content. The paper acknowledges that "through the answer channel rather than pixel storage" is not very proper.

2.Although both the title, abstract, and introduction motivate the work by memorization in "video-language models" trained on web-scale corpora, by knowledge displacement, and by exposure of sample-specific visual information. None of the measured models is a pretrained VLM. The practically relevant setting, e.g. LoRA or full fine-tuning of a pretrained video VLM, and any visual-content storage, is not studied.

  1. Eq. 5 is an accounting identity once a valid ceiling C* is assumed. Since C* is not available, the empirical test uses the union-training ceiling estimated from the same runs (a consistency check, as the authors note), while the independent grid estimate mispredicts onset by up to two chunks.
Questions for authors

Can you apply the protocol to a pretrained video VLM (e.g. Qwen2.5-VL-3B), measuring the bits stored by LoRA or full fine-tuning on video-keyed random answers? This is the setting the introduction motivates (knowledge displacement during continued training).

Please report an affine fit C = αP − β (or the fit with the stem excluded) alongside the origin-through fit. Does a single α then describe the video and text tracks across sizes, and how large is β relative to stem size?

Can the ledger be tested against a ceiling estimated independently of the stream being evaluated, e.g. union training on a different stream of the same total entropy? Otherwise the onset agreement is circular.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bkDp: Weakness 1

Weakness 1. The answers are random text strings, and the video serves only as an index. What VidMem measures is the capacity of a small from-scratch decoder to store random key to text associations, not the memorization of video content. The paper acknowledges that "through the answer channel rather than pixel storage" is not very proper.

Answer. Thank you for this point on what is stored. VidMem measures how much answer information a video-language model can store and retrieve under a video key, with the video as the key and the answer string as the stored object. This is how such a model uses what it stores: it takes video as input and produces text. Section 3.1 now defines the measured quantity in these terms, and the sentence you quote is removed.

We added a key test that measures how much of the stored answer information depends on the video key, and we applied the protocol to the pretrained model you suggested, Appendix Tables 22 and 23.

A new key test shows that most of the pooled appearance-keyed capacity depends on the correct key. If the model ignored the key during retrieval, giving every sample another sample's key on the same weights would leave compression unchanged; the observed drop therefore demonstrates dependence on the correct key. For appearance keys the key-dependent pooled slope is 2.32 bits per parameter against 2.54 for total compression, with more than 90% of the capacity depending on the key at the three largest tested sizes, which dominate the pooled fit. Text keys depend on the key at every size and motion keys little in the pooled fit, so the test does detect storage that does not need the key where it occurs. Appendix Tables 17 and 18.

Revisions to the manuscript. Section 3.1 defines the measured quantity as the answer compression that a model achieves under a given key configuration, and a new paragraph at the end of Section 3.3 states the scope of each measured quantity. The binding-cost paragraph of Section 4.2 reports the key-dependent reading for each key type.

AuthorsOpenReview ↗

Response to Reviewer bkDp: Weakness 2 and Question 1

Weakness 2. Although both the title, abstract, and introduction motivate the work by memorization in "video-language models" trained on web-scale corpora, by knowledge displacement, and by exposure of sample-specific visual information. None of the measured models is a pretrained VLM. The practically relevant setting, e.g. LoRA or full fine-tuning of a pretrained video VLM, and any visual-content storage, is not studied.

Question 1. Can you apply the protocol to a pretrained video VLM (e.g. Qwen2.5-VL-3B), measuring the bits stored by LoRA or full fine-tuning on video-keyed random answers? This is the setting the introduction motivates (knowledge displacement during continued training).

Answer. Thank you for proposing this experiment, which we ran on the model you suggested with the vision tower frozen, 16-frame appearance keys and 64-token random answers. The protocol applies to this pretrained model. Attention adapters store about one bit per adapter parameter when answers are read over the alphabet used in training, the video selects the stored answer in every tested configuration, and every tested fine-tune raises the codelength of the base model's own captions on held-out clips, by an amount that follows the optimization setting rather than the amount stored.

All three results below are new. They measure storage, key dependence and displacement on the same fine-tuned weights.

  1. A new capacity measurement shows that attention adapters store about one bit per adapter parameter under the training alphabet. Answers can be coded over the 2,048-symbol alphabet used in training or under the model's full output vocabulary. Under the training alphabet, LoRA adapters on the attention projections reach 1.11 to 1.33 bits per adapter parameter at their best dataset size, depending on the rank, and these values are lower bounds. Under the full vocabulary they store far less, because training with the restricted softmax never penalized probability outside the alphabet; in an additional run, a rank-4 adapter trained with the full-vocabulary loss stores 1.55 Mbit under the full vocabulary, 0.84 bits per adapter parameter against 0.01 with the restricted loss. Under the training alphabet, full-rank fine-tunes of the attention projections or of the whole language model store at least 98.9% of the dataset entropy at every tested dataset size, so they are data-limited and give lower bounds rather than capacities. Appendix Table 22 and Appendix F.
  2. A new key test shows that the video selects the stored answer in every tested configuration. We gave every sample another sample's video on the same fine-tuned weights. At the first answer token, which has no prefix, the wrong video raises the codelength of the true token above the 11 bits of a uniform guess in every run that stored more than 5% of the dataset entropy. When only attention is adapted, by LoRA at any rank or by full-rank fine-tuning, these runs then lose almost all stored bits. When the MLP blocks are trained as well, by a low-rank adapter or by fine-tuning the whole language model, the true first token costs 27 to 38 bits under a wrong video, and greedy decoding without an answer prefix follows the video shown. Under teacher forcing, the rest of an answer costs little once its first tokens are given, so a wrong video costs these runs only 41 to 66 of the 704 bits per sample. Appendix Table 23.
  3. A new displacement probe shows that displacement follows the optimization setting rather than the amount stored. For every fine-tuned model of Appendix Table 22 we measured the change in the base model's codelength of its own captions on held-out natural clips. Every fine-tune raised it, including one that stored almost nothing. At equal stored bits, a lower learning rate reduced the increase by an order of magnitude for attention fine-tuning and about sixfold for the whole language model, and a fine-tune on natural captions displaced the model's own captions as much as the random-code fine-tune with the same adapted modules, dataset size, learning rate and number of training steps. Appendix Tables 24 and 25.

Revisions to the manuscript. A new pretrained-model paragraph in Section 4.2 reports results 1 to 3, and Appendix Figure 10 shows captions of held-out clips after fine-tuning. Appendix F gives the runs of Appendix Tables 22 to 25 in full.

AuthorsOpenReview ↗

Response to Reviewer bkDp: Weakness 3 and Question 3

Weakness 3. Eq. 5 is an accounting identity once a valid ceiling C* is assumed. Since C* is not available, the empirical test uses the union-training ceiling estimated from the same runs (a consistency check, as the authors note), while the independent grid estimate mispredicts onset by up to two chunks.

Question 3. Can the ledger be tested against a ceiling estimated independently of the stream being evaluated, e.g. union training on a different stream of the same total entropy? Otherwise the onset agreement is circular.

Answer. Thank you for proposing this test, which we ran. Eq. 5 is a bound conditional on the unknown ceiling C★, as Section 3.3 states. What we test empirically is an onset rule whose reference is estimated on other streams: a realization rate calibrated on the submitted streams predicted the onset of all ten new streams trained with the same recipe, including four chosen so that it and the grid estimate disagree.

The original submission's test used the realized retention maximum of the same stream, a consistency check. The new test fixes ρ̄ as the mean over the six submitted streams of the realization rate ρ = C_stream/Ĉ, the realized retention maximum C_stream over the grid estimate Ĉ, and predicts the first lossy chunk of each new stream from ρ̄ and the grid estimate as k̂ = ⌊ρ̄Ĉ/H⌋ + 1, where H is the answer entropy of one chunk. No retention measured on the new stream enters the prediction, and its observed onset is read from the retention of each chunk with a fixed threshold of 0.96.

  1. Six new streams with fresh key and answer seeds show that the calibrated rule predicts every onset. At three model sizes the rule predicts all six onsets, whereas the unscaled grid estimate predicts three. Appendix Table 21.
  2. Four pre-registered streams show that the rule holds where it and the grid estimate disagree. The rule, the predictions and the onset criterion were recorded before training, and the streams were chosen so that the two rules predict different chunks. Every onset fell at the chunk the rule predicted, and the grid estimate predicted a later onset in every case. Appendix Table 21.

When the grid estimate alone misses, it predicts a later chunk, because union training realizes about 80% of it, and the calibrated rate corrects this.

Revisions to the manuscript. Section 3.3 states the ledger as conditional on C★, keeps Ĉ as a lower bound and renames C_stream the realized retention maximum, which is no longer called a ceiling. The onset paragraph of Section 4.2 reports both tests, and Appendix Table 21 lists every stream.

Reviewer_UqsjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces VidMem, a protocol that pairs video keys with independent random answers to measure the memorization capacity of video-language models. Across the tested GPT-2-style decoder grid, the pooled capacity slopes are 2.54 bits/parameter for both video tracks and 3.25 bits/parameter for the text track. Their ratio gives a cross-modal binding cost of about 1.28×. These estimates are lower bounds under the tested configurations. The paper also extends the protocol to natural videos, different frozen pretrained encoders, a second decoder family, and MoE models. It uses a capacity ledger to relate capacity to forgetting during continual learning.

Critical strengths
  1. The paper provides a unified and practical protocol for measuring visual memorization capacity.

  2. The experiments cover a broad range of settings, include substantial validation, and provide detailed procedures for reproducibility.

Critical weaknesses
  1. The paper’s most prominent result is that video-keyed associations require about 1.28× as many parameters as text-keyed associations. The authors ablate perception parameters and key length, but do not systematically study how video duration, frame count, and the complexity of video content affect this cost. The explanation of why video-keyed associations require more parameters, and when this result holds, remains incomplete.

  2. The authors use the maximum retention achieved by union training on a stream as the capacity reference, then check whether forgetting occurs when the written entropy exceeds it. This reference is only available after running the stream. The independently measured grid capacity predicts forgetting onset up to two chunks late. The paper therefore has not shown that its capacity measurements can reliably predict when forgetting will begin.

Questions for authors
  1. Beyond the existing ablations, can the authors study how video duration, frame count, and the complexity of video content affect the cross-modal binding cost? Does the 1.28× cost still hold when these factors change? Can the authors provide a more detailed explanation of the mechanisms behind the additional parameter cost of video-keyed associations?

  2. Can forgetting onset be predicted in advance without using union-training results from the target stream to set the capacity reference? If not, what new testable predictions does the capacity ledger provide beyond a post-hoc analysis of the same stream?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Uqsj: Weakness 1 and Question 1

Weakness 1. The paper's most prominent result is that video-keyed associations require about 1.28× as many parameters as text-keyed associations. The authors ablate perception parameters and key length, but do not systematically study how video duration, frame count, and the complexity of video content affect this cost. The explanation of why video-keyed associations require more parameters, and when this result holds, remains incomplete.

Question 1. Beyond the existing ablations, can the authors study how video duration, frame count, and the complexity of video content affect the cross-modal binding cost? Does the 1.28× cost still hold when these factors change? Can the authors provide a more detailed explanation of the mechanisms behind the additional parameter cost of video-keyed associations?

Answer. Thank you for these questions on what the binding cost depends on. Our ablations point to two factors behind the extra parameter cost of video keys, the parameters of the perceptual stem, which store little, and the length of the video key. At the two tested sizes, frame count changes the text-to-video ratio little, and distinctive visual cues are more common among the most-memorized natural clips.

Beyond the stem-exclusion and patch ablations of the original submission, we added the four analyses below.

  1. A new frame sweep shows that frame count changes the ratio little at both tested sizes. The 1.28× you cite is pooled over the ten model sizes of the grid at 16 frames. We added runs of the appearance track with 4, 8 and 32 frames at two of these sizes and compared them with the 16-frame runs of the original submission. Counting all parameters, the ratio moves only from 1.59 to 1.61 at the smaller size and from 1.29 to 1.34 at the larger one over an eightfold change in key tokens. With the stem excluded, the ratio is close to one at the smaller size and grows with the number of key tokens at the larger one, consistent with key length as the second factor. Appendix Table 20.
  2. A new frozen-stem control shows that the stem adds parameters but little storage. Freezing the stem, a test the original submission had left to future work, keeps 96.6% and 99.7% of the capacity at the two sizes. Counting only trainable parameters, the ratio falls from 1.60 to 1.05 and from 1.33 to 1.10. Appendix F.
  3. A new affine fit shows that the pooled ratio stays close to 1.28 when the fit includes an intercept. Fitting capacity C to parameter count P as C = αP − β, with slope α and intercept β, changes each slope by about one percent or less, and the text-to-video ratio becomes 1.26 to 1.27 instead of 1.28. Appendix Table 26.
  4. An extended analysis of natural clips shows that distinctive visual cues are more common among the most-memorized clips. We extended the analysis to the twelve most-memorized and the twelve least-memorized clips. All twelve most-memorized clips show a cut, a uniform frame or on-screen text, against five of the twelve least-memorized, and they store 3.7 times as many answer bits on average. Appendix Figure 9.

Revisions to the manuscript. The binding-cost paragraph of Section 4.2 summarizes the frame sweep and the frozen stem. Table 1 of the main text lists the affine fit, Appendix Figure 9 shows the natural clips, and Appendix F reports the other three analyses in full.

AuthorsOpenReview ↗

Response to Reviewer Uqsj: Weakness 2 and Question 2

Weakness 2. The authors use the maximum retention achieved by union training on a stream as the capacity reference, then check whether forgetting occurs when the written entropy exceeds it. This reference is only available after running the stream. The independently measured grid capacity predicts forgetting onset up to two chunks late. The paper therefore has not shown that its capacity measurements can reliably predict when forgetting will begin.

Question 2. Can forgetting onset be predicted in advance without using union-training results from the target stream to set the capacity reference? If not, what new testable predictions does the capacity ledger provide beyond a post-hoc analysis of the same stream?

Answer. Thank you for this question. We added experiments during the discussion period to answer it, and the answer is yes. Forgetting onset can be predicted before a stream is trained: a rule that uses only the grid estimate and a realization rate calibrated on other streams predicted the onset of all ten new streams trained with the same recipe, including four whose predictions were recorded before training. When the grid estimate alone misses, it predicts a later chunk, because union training realizes about 80% of it, and the calibrated rate corrects this.

The original submission compared onset with the realized retention maximum of the same stream, so the reference was available only after the run. The new test fixes ρ̄ as the mean over the six submitted streams of the realization rate ρ = C_stream/Ĉ, the realized retention maximum C_stream over the grid estimate Ĉ, and predicts the first lossy chunk of each new stream from ρ̄ and the grid estimate as k̂ = ⌊ρ̄Ĉ/H⌋ + 1, where H is the answer entropy of one chunk. No retention measured on the new stream enters the prediction, and its observed onset is read from the retention of each chunk with a fixed threshold of 0.96.

  1. Six new streams with fresh key and answer seeds show that the rule predicts every onset. At three model sizes the rule predicts all six onsets, whereas the unscaled grid estimate predicts three. Appendix Table 21.
  2. Four pre-registered streams show that the rule holds where it and the grid estimate disagree. The rule, the predictions and the onset criterion were recorded before training, and the streams were chosen so that the two rules predict different chunks. Every onset fell at the chunk the rule predicted, and the grid estimate predicted a later onset in every case. The realization rates of all ten new streams fall within the range of the submitted streams. Appendix Table 21.

Revisions to the manuscript. The onset paragraph of Section 4.2 reports both tests, and Appendix Table 21 lists every stream with its own realization rate. Beyond the points you raised, Section 4.1 merges the setup and implementation details and adds citations for the decoder, the optimizer and standard video-language corpora and benchmarks; Figures 1 and 2 of the main text are refined; Appendix Figure 9 now shows the most- and least-memorized anchor clips with their ranks; Appendix F collects the analyses run during the discussion period.

Reviewer_m6cfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VidMem, a protocol for measuring the parametric memorization capacity of video-language models in bits. It binds procedurally generated video keys to independent random answer strings of known entropy and measures memorized information through the reduction in conditional codelength relative to a uniform reference.

Critical strengths
  • The paper introduces a clear information-theoretic unit for studying video-conditioned memorization. The random-assignment construction separates sample-specific storage from semantic generalization.
  • The experimental study including multiple model sizes, appearance and motion keys, natural-video anchors with frozen encoders, a second decoder family, mixture-of-experts models, precision controls, and stability analyses.
Critical weaknesses
  1. The scope and practical relevance of the measured memorization should be clarified.

    • VidMem measures random associations keyed by video identity, rather than memorization of the video content itself.The random-answer setting deliberately forces memorization and evaluates the same training associations, but does not show how much of this capacity is used during natural VLM training or whether it benefits real tasks.
    • The paper should distinguish more clearly between memorization capacity, realized memorization, and task utility, and narrow its practical claims accordingly.
  2. The measured capacity is a lower bound rather than the ceiling required by the forgetting analysis.

    • The reported value is the best result under finite searches over dataset sizes, optimization settings, checkpoints, and training budgets, so it only lower-bounds attainable capacity. The forgetting ledger requires a valid upper bound, while CstreamC_{\text{stream}} is only the best retention observed from union training on the same stream.
    • The terminology should be revised, and a capacity estimated on one stream should ideally be tested for its ability to predict forgetting onset on independent streams.
  3. The motion track does not isolate temporal-order memorization.

    • Frame shuffling changes the measured capacity by at most 0.9%, indicating that motion keys remain identifiable from the unordered set of object positions. The track therefore measures trajectory-derived visual statistics rather than temporal-order-dependent binding.
    • A cleaner test would use videos with identical frame multisets but different frame orders, or otherwise weaken the claims about temporal memorization.
Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer m6cf: Weakness 1

Weakness 1. The scope and practical relevance of the measured memorization should be clarified. VidMem measures random associations keyed by video identity, rather than memorization of the video content itself. The random-answer setting deliberately forces memorization and evaluates the same training associations, but does not show how much of this capacity is used during natural VLM training or whether it benefits real tasks. The paper should distinguish more clearly between memorization capacity, realized memorization, and task utility, and narrow its practical claims accordingly.

Answer. Thank you for drawing these distinctions, which the revised version now makes explicit. VidMem measures a lower bound on memorization capacity, the answer information that a fixed model, precision and answer format can store, with the video as the key and the answer string as the stored object. Expressed in bits per parameter, it compares storage across model sizes, architectures and key types on one scale. Scaled by a realization rate calibrated on other streams, it predicted when forgetting began on all ten new streams trained with the same recipe, as Appendix Table 21 shows. Realized memorization in natural training and task utility are different quantities that the random-answer construction excludes by design, and Section 3.3 of the revised version now states this scope.

Revisions to the manuscript. Section 3.1 defines the measured quantity as the answer compression a model achieves under a given key configuration. A new paragraph at the end of Section 3.3 separates memorization capacity from realized memorization and task utility, states that VidMem measures neither of the latter, and restricts forgetting to stored answer information.

AuthorsOpenReview ↗

Response to Reviewer m6cf: Weakness 2

Weakness 2. The measured capacity is a lower bound rather than the ceiling required by the forgetting analysis. … The forgetting ledger requires a valid upper bound, while C_stream is only the best retention observed from union training on the same stream. The terminology should be revised, and a capacity estimated on one stream should ideally be tested for its ability to predict forgetting onset on independent streams.

Answer. Thank you for this point, which the revised version takes up in its terminology and in a new prediction test. The revised version separates three quantities: C★ is the unknown ceiling that Eq. 5 requires, Ĉ is a finite-grid lower bound, and C_stream is renamed the realized retention maximum of one training recipe on one stream. The ledger is stated as conditional on C★, and we ran the test you suggest on independent streams: the grid estimate, scaled by a realization rate calibrated on the submitted streams, predicted the onset of forgetting on all ten new streams trained with the same recipe.

The original submission compared the onset of forgetting with C_stream of the same stream, which is a consistency check. The new test fixes ρ̄ as the mean over the six submitted streams of the realization rate ρ = C_stream/Ĉ and predicts the first lossy chunk of each new stream from ρ̄ and the grid estimate as k̂ = ⌊ρ̄Ĉ/H⌋ + 1, where H is the answer entropy of one chunk. No retention measured on the new stream enters the prediction, and its observed onset is read from the retention of each chunk with a fixed threshold of 0.96.

  1. Six new streams with fresh key and answer seeds show that the calibrated rule predicts every onset. At three model sizes the rule predicts all six onsets, whereas the unscaled grid estimate predicts three. Appendix Table 21.
  2. Four pre-registered streams show that the rule holds where it and the grid estimate disagree. The rule, the predictions and the onset criterion were recorded before training, and the streams were chosen so that the two rules predict different chunks. Every onset fell at the chunk the rule predicted, and the grid estimate predicted a later onset in every case. The realization rates of all ten new streams fall within the range of the submitted streams. Appendix Table 21.

Revisions to the manuscript. The introduction and Section 3.3 distinguish the three quantities, and C_stream is no longer called a ceiling. The onset paragraph of Section 4.2 reports both tests, and Appendix Table 21 lists every stream with its own realization rate.

AuthorsOpenReview ↗

Response to Reviewer m6cf: Weakness 3

Weakness 3. The motion track does not isolate temporal-order memorization. Frame shuffling changes the measured capacity by at most 0.9%, indicating that motion keys remain identifiable from the unordered set of object positions. The track therefore measures trajectory-derived visual statistics rather than temporal-order-dependent binding. A cleaner test would use videos with identical frame multisets but different frame orders, or otherwise weaken the claims about temporal memorization.

Answer. Thank you for this careful reading of the frame-shuffling result. The revised version describes the motion track as storage keyed by trajectory-derived visual statistics and no longer claims that it measures temporal-order binding. The reported motion-track capacities are unchanged, since only their interpretation is narrowed.

This description agrees with the frame-shuffling ablation you cite, which Section 4.3 and Appendix D report: at three model sizes, shuffling frame order changes the capacity of either video track by at most 0.9%.

Revisions to the manuscript. Section 3.2 now says that motion keys differ only in trajectory-derived visual statistics and no longer says that separating them requires temporal integration.

Other revisions. Section 4.1 merges the setup and implementation details and adds citations for the decoder, the optimizer and standard video-language corpora and benchmarks; Figures 1 and 2 of the main text are refined; Appendix Figure 9 now shows the most- and least-memorized anchor clips with their ranks; Appendix F collects the analyses run during the discussion period.

Reviewer_2F4TOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work is inspired by the memorization framework developed by Morris et al. in the text space, where they were able to give an account of the number of bits of information memorized by pre-trained models. They want to extend this to video language models.

For this purpose, we have a synthetic video setup, and the authors try to correlate these videos with entirely random answers in the dataset. Memorization over here is being measured based on how much we can reduce the code length of the answers as compared to a uniform baseline.

The headline number of the paper says that 2.54 bits per parameter in video-conditioned associations and 3.25 bits in tax-condition associations are seen, which suggest a 1.28x cost in the parameter space for video keys. The work then goes into understanding and analyzing various different effects within architectural choices of the model.

Critical strengths

The main strength of this paper is that it takes a well-defined framework and consolidates it into a nice experimental framework in a new domain. Based on the construction of these answers, you can clearly tell the effect of memorization across different aspects of model training, like architecture, data, and different modalities.

Secondly, as a consequence of this framework, the authors were able to produce some interesting observations around the configurations of representations and so on.

Critical weaknesses

My biggest concern is that I am unconvinced that this measure is able to isolate the quantity that the work claims, which is video-condition memorization. There is a big distinction between the work of Morris et al. and VidMem. In the prior work, which was inspiration for VidMem, the random sequences were the quantity of interest and were being memorized. In this case, we have random video keys, which are being associated with random answers, and we are measuring the likelihood of the answer as a proxy for memorization.

The challenge is that when you're making the next token, because the answer can be long, the previous correct answer tokens are actually already available. In this case, since the answer is, say, 64 tokens on a certain vocabulary size, even with small answer prefixes, we can almost uniquely identify which sequence was used. This really means that a model may not even need to make any associations with the video, and it may just be relying on the text itself. In fact, there is already an ablation that this paper does: for larger models, there is a keyless baseline where you don't even have the video key, which can achieve substantial memorization. We are not really able to attribute the information to just video-answer association.

My second concern is that it's unclear whether the scientific contributions on top of the previous work are tall enough and novel enough. Even if it's not novel, I would have loved a more principled understanding of the findings in this case. Because of the previous concerns in one, I don't think this framework is ready for this.

Questions for authors

Given the keyless baseline shows that we can already have a lot of association with random answers, how can we say that the 2.54 bits per parameter is the measurement of video-conditioned associative storage rather than just on the answer sequence?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2F4T: Weakness 1 and Question

Weakness 1. … The challenge is that when you're making the next token, because the answer can be long, the previous correct answer tokens are actually already available. In this case, since the answer is, say, 64 tokens on a certain vocabulary size, even with small answer prefixes, we can almost uniquely identify which sequence was used. This really means that a model may not even need to make any associations with the video, and it may just be relying on the text itself. … We are not really able to attribute the information to just video-answer association.

Question. Given the keyless baseline shows that we can already have a lot of association with random answers, how can we say that the 2.54 bits per parameter is the measurement of video-conditioned associative storage rather than just on the answer sequence?

Answer. Thank you for this question, which goes to the core of what VidMem measures. The 2.54 bits per parameter measure total answer compression, and the revised version measures the key-dependent part of it directly on the same weights and reports it beside the total. For appearance keys most of the pooled capacity depends on the correct key, more than 90% at the three largest tested sizes, which dominate the pooled fit.

The keyless baseline you cite, in Table 1 of the main text, is a separate model and cannot show which channel a keyed model uses. During the discussion period we therefore added analyses of the submitted model weights and one new experiment, each designed to test one way the prefix could explain the measured capacity.

  1. A new key test shows that most of the pooled appearance-keyed capacity depends on the correct key. If the model ignored the key during retrieval, giving every sample another sample's key on the same weights would leave compression unchanged; the observed drop therefore demonstrates dependence on the correct key. For appearance keys the key-dependent pooled slope is 2.32 bits per parameter, against 2.54 for total compression, and more than 90% of the capacity depends on the key at the three largest tested sizes, which dominate the pooled fit. Text keys depend on the key at every size and motion keys little in the pooled fit, so the test does detect storage that does not need the key where it occurs. A blank clip in place of the key removes even more compression, but it is an out-of-distribution input, so we use the more conservative permuted key as the primary diagnostic. Appendix Tables 17 and 18.
  2. A new per-position analysis shows that below capacity the key supplies the first answer token. The first answer token has no prefix, so whatever a keyed model stores there beyond what a model without the key can store must come from the key. We derive the exact maximum that any model without the key can store at this position. At the dataset sizes of the keyed runs below capacity it is at most 1.2 of the 11 bits, while these keyed models store 10.3 to 11.0 bits there. Appendix Tables 16 and 27.
  3. A new short-answer experiment shows that the key saves the cost of selecting one answer among N. Without a key, a model must spend on average about log₂N bits per answer to select one of the N stored strings, and a key can remove this cost. We retrained with 4-token answers, where this cost is a larger fraction of the codelength. At N = 32,768, below capacity, both keyed tracks store almost all 44 bits, about 15 more than the keyless baseline, and a wrong key leaves almost nothing stored. Appendix Table 19.

Revisions to the manuscript. Contribution 1 and the capacity paragraph of Section 4.2 present 2.54 as total answer compression, and a new second contribution introduces the key test. The binding-cost paragraph of Section 4.2 reports the key-dependent reading for each key type. Appendix F gives every result in full.

AuthorsOpenReview ↗

Response to Reviewer 2F4T: Weakness 2

Weakness 2. My second concern is that it's unclear whether the scientific contributions on top of the previous work are tall enough and novel enough. Even if it's not novel, I would have loved a more principled understanding of the findings in this case. Because of the previous concerns in one, I don't think this framework is ready for this.

Answer. Thank you for raising novelty and the wish for a more principled understanding. VidMem is, to our knowledge, the first protocol that measures the memorization capacity of video-language models in bits per parameter. It is new in three respects: video keys are paired with random answers of known entropy, so stored answer information is measurable without a video likelihood; a key test measures how much answer compression depends on the correct video key; and an onset rule calibrated on other streams predicts when forgetting starts on a new stream before it is trained. The revised version adds the key test and the onset rule to the contribution list of the introduction. It also derives, with no fitted quantity, an exact ceiling on how much of a stored answer can be retrieved without the key, and models trained without a key come within 0.4 bit of this ceiling when well below capacity. Each contribution is listed below.

  1. The original submission introduced video keys paired with random answers of known entropy. The lattice construction of Section 3.2 makes the keys distinct and exactly reproducible, and the known answer entropy lets us measure stored answer information without computing video likelihoods.
  2. A new key test measures how much answer compression depends on the correct video key. We give every sample another sample's key on the same weights and measure how much compression remains. For appearance keys the key-dependent pooled slope is 2.32 bits per parameter against 2.54 for total compression, and more than 90% of the capacity depends on the key at the three largest tested sizes. Appendix Tables 17 and 18.
  3. New controls extend the original submission's account of what contributes to the extra parameter cost of video keys. The original submission associated part of the cost with the perceptual stem and key length through its stem-exclusion and patch ablations. A new frozen-stem control barely changes capacity, and a new sweep from 4 to 32 frames shows the residual cost without the stem growing with the number of key tokens at the larger size tested. Appendix Table 20 and Appendix F.
  4. The bits-per-parameter unit extends beyond one architecture. Mixture-of-experts capacity follows total rather than active parameters, Appendix Table 15, and five frozen encoders give similar estimates once the projection layer is excluded, Table 2 of the main text.
  5. An onset rule calibrated on the submitted streams predicts when forgetting starts on new streams. The original submission checked the ledger against a reference taken from the same stream. The rule and its predictions were recorded before training, on four new streams of the same recipe, chosen so that the rule and the grid estimate disagree. In all four, forgetting began at the chunk the rule predicted, and the grid estimate predicted a later onset in every case. Section 4.2 and Appendix Table 21.
  6. A new derivation gives an exact ceiling on retrieval without the key, and models trained without a key reach it to within 0.4 bit when well below capacity. We prove that the measured stored bits are, in expectation and up to one bit per answer, a lower bound on the answer information held in the weights, whichever input retrieves it, and that for the N distinct answers of K bits each in our datasets, no predictor without the key can store more than K − log₂N bits per answer on average. The bound holds position by position and contains no fitted quantity. In the five runs of Appendix Table 27 that are furthest below capacity, models trained without a key reach it to within 0.4 bit in total and 0.2 bit at each position. Below capacity the keyed models of Appendix Table 16 exceed it by 3.6 to 10.4 bits. Appendix F and Appendix Table 27.

The new controls link the extra parameter cost of video keys to the perceptual stem and key length, the key test shows how much answer compression depends on the correct video key, and the derivation gives the exact limit of retrieval without the key.

Presentation. We have revised the presentation. The revised version adds a paragraph at the end of Section 3.3 that states the scope of each measured quantity, extends the binding-cost paragraph of Section 4.2 with the distinction between total and key-dependent storage, and refines Figures 1 and 2 of the main text.

Other revisions. Section 4.1 merges the setup and implementation details and adds citations; Appendix Figure 9 now shows the most- and least-memorized anchor clips with their ranks; Appendix F collects the analyses run during the discussion period.

AuthorsOpenReview ↗

Follow-up to Reviewer 2F4T

Thank you again for your review. As the discussion period closes in a few hours, we would like to ask whether our responses have addressed your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.