Where Does the Video Go? Probing Persistent Temporal Memory after Visual KV Eviction in Video LLMs

Video large language models (video LLMs) represent a video as a long multimodal prefix, producing large key–value (KV) caches that can remain resident when the same video is reused for multiple temporal queries. Existing efficiency methods primarily reduce visual computation before or during multimodal processing, or compress the resulting cache by retaining entries selected using importance or relevance scores. What remains unclear is which KV states must stay resident once the complete video prefix has already participated in multimodal prefill. We study this post-prefill setting, in which the complete video prefix is processed once and its KV cache is physically compacted before any downstream query is observed. Across three video LLM families, we find that substantial temporal-grounding ability remains after most visual-payload KV states are removed when the compact cache retains sparse temporal-carrier positions aligned with each model's video serialization. Under identical physical KV budgets, retaining these carrier positions consistently outperforms replacing them with additional visual-payload states, showing that post-prefill memory is functionally organized rather than uniformly distributed across the video prefix. Controlled carrier-content interventions further show that the retained states carry information tied to the processed video, rather than acting only as generic positional scaffolds. Motivated by these findings, we introduce TC-KV, a simple, training-free, and query-independent cache policy that preserves the temporal-carrier structure together with a small visual residual after complete prefill. TC-KV thereby substantially reduces persistent video-prefix KV and enables the same compact memory to serve multiple temporal queries, without retraining or modifying the complete multimodal prefill.