acceptodds
Under review as a conference paper at ICLR 2027

VideoLoop: Looped Working Memory Against Semantic Thrashing in Long-Form Video Agents

Abstract

Long-form video understanding requires multimodal agents to iteratively gather evidence over many reasoning steps. However, most existing agentic methods suffer from *semantic thrashing*: as append-only working memory grows, attention to key evidence collapses, and the agent loses access to what it has already found. First, we provide a structural argument showing that append-only memory can incorporate newly observed target evidence but cannot remove accumulated noise or prevent the growth of ordered context without a rewrite operator. Second, motivated by this analysis, we propose VideoLoop, a multimodal agent with two coupled loops. The outer loop reasons over the video and the inner loop, after each step, retrieves artifacts from an unbounded filesystem of past observations and intermediate analysis, and rewrites a bounded working memory. Extensive experiments demonstrate the effectiveness of VideoLoop, which improves four popular LVLM backbones in a plug-and-play manner, with an average gain of 4.2% points over baseline on VideoMME (*long*). Further analysis of working memory suggests that VideoLoop mitigates semantic thrashing, achieving 81% context-only judge accuracy on a human-reviewed challenging subset versus 59% for the append-only agent. Notably, with Gemini 3.1 Pro, our VideoLoop achieves new state-of-the-art results on all three benchmarks: 88.3% on VideoMME (long), 88.8% on VideoMMMU, and 80.9% on LongVideoBench (long).

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-form Video Understanding, Multimodal Agents, Working Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5mtMOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VideoLoop addresses the difficulty of retaining usable evidence during long video-agent trajectories, described as semantic thrashing. An outer multimodal policy explores the video with tools and stores observations in a persistent filesystem. After each step, a separate inner orchestrator retrieves relevant artifacts and edits a six-section working memory with a target token budget. A set-based analysis motivates deletion and recovery of evidence. Evaluation covers VideoMME (long), VideoMMMU, and LongVideoBench (long), with positive reported gains across four policy backbones. The paper also compares memory configurations, measures context-only question answerability, and introduces fact-level retention probes. The strongest empirical contribution is the combined system's accuracy improvement together with a more direct evaluation of whether previously recovered facts remain usable.

Critical strengths
  1. The retrieval-and-rewrite design has useful empirical support. Preserving raw artifacts while editing the policy's current memory provides a practical way to revisit evidence. In Table 2, the Gemini 3 Flash comparison reports 81.9% for append-only, 83.3% for rewriting without filesystem retrieval, and 85.8% for the full system. The Q4 improvement from 70.7% to 80.0% is particularly relevant to the intended difficult questions. These are single-run results, but the component comparison, three-benchmark results in Table 1, and four-backbone results in Figure 3 collectively support the utility of this system combination.

  2. The additional retention study provides information beyond final task accuracy. Appendix F evaluates 3,387 probes covering 300 videos and 900 questions, uses a different model family as the reader, hides probe answer options, and reports video-cluster bootstrap intervals. Among probes recovered at least once by each method, the final kept rates are 88.7% for VideoLoop and 65.9% for append-only (Figure 6). Separating initial recovery from subsequent availability is consequential for understanding memory behavior. The manually constructed probe subset and the stated caveats also make this evidence more informative than the headline answerability curve alone.

Critical weaknesses
  1. The retention comparison does not isolate the proposed semantic-thrashing mechanism. Appendix F limits the append-only query plus history to 32K tokens and trims excess history, whereas VideoLoop's 32K limit applies to working memory before adding the query and recent interaction window. A matched total input budget is therefore not established. The appendix also acknowledges that it cannot separate transcript removal from difficulty recovering evidence that remains in context. These results support better evidence availability under the tested configurations, but leave the attention-dilution explanation in Sections 3.1 and 4.4 unresolved. This distinction matters because diagnosing and mitigating that failure mode is a central scientific claim. The acknowledged truncation and budget scope need to be reflected in that claim.

  2. The fixed-context guarantee is not established for complete loop inputs. Section 3.3 concludes that the outer context has a constant bound, but limiting the recent window to eight triplets does not bound their token lengths. Tool observations can contain a full transcript or variable-length code output (Section 3.2). The inner loop also reads the manifest, filesystem index, and retrieved files without a stated total input limit. Algorithm 1, step 15, labels the memory limit a target; Appendices H and J do not specify token validation and oversize handling. Consequently, history growth could be shifted into the orchestrator, which would undermine the central scalability argument. Existing enforcement rules and input-length logs could resolve this concern; otherwise, the guarantee should be restricted to the components actually bounded.

Questions for authors
  1. Existing configuration and log clarification. What hard limits and oversize-handling rules apply to both loops' complete inputs, including recent observations, manifest summaries, the filesystem index, retrieved files, and images? Please use existing specifications and length logs to explain the fixed-bound claim and whether Appendix F used equal total policy/judge budgets.

  2. Suggested limited analysis. Could a small, manually checked subset of existing Appendix F snapshots distinguish facts removed by trimming from facts still present but no longer recovered? This would be additional analysis of existing runs. If unavailable, can the recognized truncation confound be reflected in the mechanism claim?

  3. Aggregation clarification. Under Appendices D/E's 225-question, unsmoothed binary final-snapshot protocol, no integer count yields 81.1%; 182/225 and 183/225 round to 80.9% and 81.3%. What numerator, denominator, or averaging rule was actually used? Please correct the value or protocol as appropriate. The neighboring counts would preserve nearly all of the reported advantage.

If my concerns are adequately addressed, I would be willing to raise my score.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5mtM (Part 1/3: Q1 Budgets and Retention)

Q1: Were total policy input budgets matched in Appendix F?

A1: No. The original 32K budgets covered different prompt components. Append only limited the question and history together to 32K tokens. VideoLoop applied the 32K limit to working memory, with the other prompt components added separately.

To test whether more input capacity removes the retention advantage, we have repeated the experiment with a 60K append only budget. We have also repeated the Appendix F evaluation. The 60K budget exceeds VideoLoop's largest recorded complete policy input of 58.4K.

The table tracks fact questions, or probes, with annotated answers. It reports the percentage of previously recovered facts that the reader cannot recover at each stage. For the final step, each method's denominator is all probes recovered at least once. Final loss is therefore 100% minus final retention.

MethodBudgetMax input per call0 to 25%25 to 50%50 to 75%75 to 100%Final step
Append only, originalQuery + history ≤ 32K32.0K6.2%16.6%28.5%35.5%34.1%
Append only, newQuery + history ≤ 60K60.0K10.7%14.3%15.6%23.7%25.1%
VideoLoopWorking memory ≤ 32K58.4K0.9%6.0%9.3%11.2%11.3%

VideoLoop reduces final evidence loss to 11.3%, compared with 25.1% for the 60K baseline, a difference of 13.8 percentage points. Its advantage holds at every reported stage. We will add this comparison and the denominator definition to Section 4.4 and Appendix F.

Q1.a: Were total judge input budgets matched in Appendix F?

A1.a: No. Each method provides its full available memory snapshot, so reader input lengths differ. A snapshot is the memory saved at a particular reasoning step. All methods use the same reading protocol, without extra truncation and within the provider's context limit.

A1 addresses the policy input budgets. Here, we evaluate the memory retained by each method. The reader tests whether it can recover facts from the memory that each method retains.

Across the 3,387 fact probes with answer options hidden, VideoLoop retains 88.7% of previously recovered facts. Append only retains 65.9% with a 32K budget and 74.9% with a 60K budget. A1.b breaks down the final outcomes.

Q1.b: Can existing snapshots distinguish trimmed facts from facts that remain present but become unrecoverable?

A1.b: Partly. The check identifies cases where the answer text remains in the snapshot but the reader cannot recover the answer. If the text is absent, it can have been trimmed or rewritten in different words.

To distinguish these outcomes, we have searched each final snapshot for the annotated answer and its accepted alternative forms. The table separates successful recovery, failure with no text match, and failure with a text match.

Final outcomeAppend only, 32KAppend only, 60KVideoLoop
Recovered65.9%74.9%88.7%
Not recovered, no answer string match30.5%23.0%8.5%
Not recovered, answer string matched3.6%2.1%2.8%

Each column counts probes recovered at least once by that method. VideoLoop reduces unrecovered probes with no answer text match from 23.0% for the 60K baseline to 8.5%.

Q1.c: How should the truncation confound limit the semantic thrashing claim?

A1.c: Under bounded budgets, append only loses recoverable facts mainly through history trimming once raw observations hit the cap, alongside context dilution on retained text.

Even at a 60K budget exceeding VideoLoop's 58.4K maximum, append only hits the cap and loses 25.1% of recovered facts versus 11.3% for VideoLoop.

We will revise Sections 3.1 and 4.4 to scope our claim around joint trimming and dilution under bounded context, rather than attention dilution alone.

(Continued in Part 2/3)

AuthorsOpenReview ↗

Response to Reviewer 5mtM (Part 2/3: Q2 and Q2.a Loop Limits)

Q2: What hard token limits apply to the complete outer loop input?

A2: The 32K token limit applies to working memory. The complete outer request is assembled from the components below and has no separate total token check. The table lists the actual limits and handling rules for each component.

Input componentBudget or handling
Stored working memoryBudget: 32K tokens. Oversized memory is shortened by section after an update, followed by text truncation if needed
Recent interactionsRetain the most recent 8 turns, with character clipping of policy text
Requested transcriptRetain one transcript block outside working memory
System instructions, query, and tool definitionsAdd during request assembly
Video frame analysisanalyze_frames splits requested frames into batches of at most 64. Each analysis request keeps the latest image message. Subsequent ordinary policy calls remove image blocks from multimodal user messages
Question imagesAttach during initial request construction, without a separate image count cap

The largest recorded complete policy input is 58.4K tokens. When frame analysis uses the agent API, its input count includes image tokens. We include these calls in agent usage.

Q2.a: What hard token limits apply to the complete inner loop input?

A2.a: The inner loop limits text length, frame counts, and tool rounds. The table specifies what the memory orchestrator can receive. The orchestrator is the model responsible for retrieving evidence and rewriting working memory.

ComponentImplementation rule
Filesystem contextClip the assembled view to 16,000 characters
Step recordsTarget: 16,000 characters. Keep at least two records and a marker for omitted content, which can extend the text beyond this target
New tool observationClip to 16,000 characters
Attached policy reasoningClip to 4,000 characters
File retrievalread_file defaults to 4,000 characters and permits up to 8,000. Returned tool text is clipped to 4,000 on insertion
Frame inspectionEach frame inspection samples at most 6 frames
Frame selectionFrame selection samples at most 30 candidates and selects one for working memory
Images in rewrite callsInitially attach question images. Before subsequent tool rounds, remove image blocks from messages preceding the last two messages
Tool roundsAt most 5 per rewrite

The largest recorded complete inner call contains 10,510 input tokens. This count includes instructions, previous memory, accumulated tool exchanges, and image tokens reported by the provider.

(Continued in Part 3/3)

AuthorsOpenReview ↗

Response to Reviewer 5mtM (Part 3/3: Q2.b, Q2.c, and Q3)

Q2.b: How are oversized inputs handled in each loop?

A2.b: Oversized working memory is shortened by section, then truncated if it remains over the 32K budget. Retrieved text and other inner inputs are clipped before use. The table gives the handling rules.

ContentHandling
Oversized working memoryKeep the last 10 nonempty lines of Key Evidence, shorten Current Understanding to 3 sentences, then truncate excess text
Filesystem viewKeep the beginning of the text, up to 16,000 characters
Retrieved file textRead 4,000 characters by default, up to 8,000. Clip the returned text to 4,000 characters when adding it to the inner conversation
Step recordsTarget: 16,000 characters. Keep at least two records and a marker for omitted content, which can extend the text beyond this target
Video frame analysisSplit requested frames into batches of at most 64 and retain the latest image message. Ordinary policy calls then remove image blocks from multimodal user messages
Inner visual callsEach frame inspection samples at most 6 frames. Frame selection samples at most 30 candidates
Images in rewrite callsBefore later tool rounds, remove image blocks from messages preceding the last two messages
TranscriptRetain one transcript block outside working memory
60K append only controlRemove older messages until the question and history fit within the input budget

Q2.c: Can the input length logs establish a fixed context limit?

A2.c: No. The logs establish observed input sizes. The largest recorded inputs are 58.4K tokens for the outer loop and 10,510 for the inner loop. To examine how input size changes during a run, we group the calls by reasoning step.

Each cell reports the median and maximum input tokens per call.

StepsOuter, VideoLoopOuter, append onlyInner, VideoLoop
1 to 517.7K / 44.4K10.4K / 60.0K3.6K / 10.5K
6 to 1523.0K / 58.4K16.3K / 60.0K5.1K / 10.1K
16 to 50N/A7.6K / 60.0KN/A

Across these logged runs, all VideoLoop trajectories finished within 15 steps under the shared 50 step limit. We will revise Section 3.3 and Algorithm 1 to restrict our formal bound claim to the explicitly bounded components and report measured complete request sizes.

Q3: How was 81.1% computed under the 225 question, binary final snapshot protocol?

A3: The original 81.1% was 180/222. We now count the three runs that ended with errors as incorrect, giving 180/225 = 80.0%. Thank you for identifying the denominator issue.

Method and calculationCorrect answersQuestions countedAccuracy
VideoLoop, original calculation18022281.1%
VideoLoop, corrected calculation18022580.0%
Append only13722560.9%

Across all 225 questions, VideoLoop achieves 80.0% accuracy versus 60.9% for append only, a 19.1 percentage point advantage. We will update Appendices D and E with the full denominator and corrected score.

Reviewer_xJf3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VideoLoop separates video exploration from memory management. An outer agent gathers evidence, while an inner orchestrator retrieves stored artifacts and rewrites a bounded working memory after each step. The method requires no additional training. Experiments across several backbones report improved video QA accuracy, and context-based probes examine whether previously acquired evidence remains recoverable.

Critical strengths
  1. Controlled memory ablation. With the same backbone, VideoMME-Long accuracy rises from 81.9% for append-only memory to 83.3% for rewrite-only memory and 85.8% for the full method (Table 2). Total token use stays close to append-only, although the increased output-token share can raise monetary cost (Table 3; Appendix A).
  2. Training-free applicability. Figure 3 reports improvements over native inference for four policy backbones without additional parameter training. This supports the usefulness of memory orchestration beyond a single model and avoids training a separate memory policy.
Critical weaknesses
  1. Practical serving cost is not quantified. The orchestrator runs after every outer step, and output tokens roughly double. Appendix A acknowledges the resulting monetary-cost increase, but end-to-end latency and actual API cost are not reported. Similar total token counts therefore do not establish similar serving cost.
  2. Some design choices need clearer justification. The framework is well motivated, but the ablations do not establish whether per-step orchestration is necessary, or whether periodic or condition-triggered updates could preserve accuracy at lower cost. The working-memory cap and manifest compression rules are specified, but the size limits for the orchestrator's full input are less clear as the artifact store grows.
Questions for authors
  1. What are the end-to-end latency and monetary cost per question for the memory variants, including policy, tool-analysis, and orchestrator calls? Please specify the model and pricing assumptions.
  2. Why invoke the orchestrator after every outer step rather than every few steps or when new evidence or memory pressure warrants an update? Could these alternatives preserve accuracy with fewer orchestration calls?
  3. Beyond the 32K working-memory cap and manifest compression rules, what limits apply to the filesystem index and retrieved file contents? How is the orchestrator's total input kept within its context budget?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xJf3 (Part 1/2: Q1 Latency and API Cost)

Q1: What is the end to end latency per question for each memory variant?

A1: VideoLoop takes 237.4 s per question, compared with 308.9 s for append only and 450.8 s for dual loop. Its total latency is 23.1% lower than append only.

The timing breakdown includes policy calls, tool analysis, memory maintenance, tool execution, container setup, and bookkeeping.

VariantAgent LLM callsOrchestrator LLM callsToolsOtherEnd to end
Append only29.8 calls, 292.4 sN/A7.3 s9.2 s308.9 s
Dual loop15.6 calls, 185.6 s10.6 calls, 182.1 s16.9 s66.2 s450.8 s
VideoLoop13.6 calls, 182.0 s7.8 calls, 27.3 s6.3 s21.8 s237.4 s

Agent calls include both policy decisions and tool analysis. VideoLoop reduces these calls from 29.8 to 13.6 per question. The Other column includes container setup and bookkeeping, where dual loop incurs higher latency from larger unclipped rewrite inputs (91.8K versus 47.7K tokens) and container queuing.

Q1.a: What is the total API cost per question, including policy, tool analysis, and orchestrator calls?

A1.a: The average API cost is $0.320 per question for VideoLoop, $0.155 for append only, and $0.324 for dual loop. The calculation includes the recorded policy, tool analysis, and orchestrator usage.

VariantInput (cache reads)OutputAgentOrchestratorTotal / question900 questions
Append only584.6K (86.9%)30.3K$0.155N/A$0.155$139
Dual loop590.0K (53.6%)57.0K$0.245$0.078$0.324$291
VideoLoop559.0K (54.6%)59.2K$0.269$0.051$0.320$288

All methods use Gemini 3 Flash. Per million tokens, the prices are $0.50 for uncached input, $0.05 for cached input, and $3.00 for output. VideoLoop's orchestrator costs $0.051 per question, or about 16% of the total.

The cost difference from append only comes mainly from output tokens and cache reuse. At an additional $0.165 per question, VideoLoop gains 3.9 accuracy points and reduces total latency by 23.1%.

We will add the cached input percentage, total latency, and API cost to Table 3 and Section 4, with pricing details in Appendix A.

Q1.b: Which models and prices are used for the cost calculation?

A1.b: All calls in the Table 3 comparison use Gemini 3 Flash. We calculate costs using the reported Google AI Studio paid tier prices for November 2026. As requested, the table specifies the price for each token category.

Token categoryRate per 1M tokens
Uncached input$0.50
Cached input$0.05
Output, including billed thinking tokens$3.00

We use logged token counts and include billed thinking tokens in output. The calculation assumes implicit caching, with no separate charge for cache writes or storage.

For example, VideoLoop uses 305.0K cached input tokens, 254.0K uncached input tokens, and 59.2K output tokens per question. Input costs $0.142 and output costs $0.178, giving a total of $0.320.

(Continued in Part 2/2)

AuthorsOpenReview ↗

Response to Reviewer xJf3 (Part 2/2: Q2 Schedules and Q3 Bounds)

Q2: Why invoke the orchestrator after every outer step?

A2: We use every step updates because they achieve the highest accuracy among the three schedules. Each update combines the latest observation with relevant earlier evidence before the next policy decision.

To test the alternatives you raise, we have compared all three schedules on 900 VideoMME long video questions using Gemini 3 Flash. This tests whether fewer memory updates preserve accuracy and reduce total cost or latency.

Update scheduleAccuracyRewrites, relative to every stepAverage cost per questionLatency per question
Every step85.8%100%$0.320237.4 s
Every two steps83.9%57%$0.322248.0 s
Triggered84.8%57%$0.313235.0 s

Triggered updates run after frame analysis, transcript access, or a large tool output, and at least once every three steps. Every step updates gain 1.0 accuracy point over triggered updates for an additional $0.007 and 2.4 s per question.

The default orchestrator costs $0.051 and takes 27.3 s per question, accounting for 16% of API cost and 11.5% of total latency. We will report the scheduling comparison in Section 4 alongside the cost and latency breakdown in Table 3.

Q2.a: Can periodic or condition triggered updates preserve accuracy with fewer orchestration calls?

A2.a: The alternatives reduce rewrites by 43%, while every step updates retain the highest accuracy. Periodic and triggered updates achieve 83.9% and 84.8%, respectively, compared with 85.8% for the default.

Triggered updates offer a lower cost option. Triggered updates cost $0.313 and take 235.0 s per question. We use every step updates as the default because an additional $0.007 and 2.4 s gains 1.0 accuracy point.

Fewer rewrites do not translate directly into the same reduction in total cost or latency. The orchestrator accounts for 16% of cost and 11.5% of latency, and total usage also includes the agent's exploration. Less frequent updates leave working memory staler and increase outer exploration, so periodic updates reach $0.322 and 248.0 s versus $0.320 and 237.4 s for every step updates.

Q3: How is the orchestrator's total input bounded as the artifact store grows?

A3: The orchestrator receives a selected filesystem view and short excerpts from retrieved files. The table lists the limits on this view, retrieved content, and each rewrite.

ComponentControl
Filesystem contextClip the assembled view to 16,000 characters
Step recordsKeep 30 detailed step records and summarize older records in groups of 10. Target length: 16,000 characters
Retrieved file contentDefault read of 4,000 characters, up to 8,000. Clip returned text to 4,000 characters on insertion
New observation and policy reasoningClip to 16,000 and 4,000 characters, respectively
Tool roundsAt most 5 per rewrite
Frame inspectionAt most 6 frames per inspection

The step record view targets 16,000 characters. It retains at least two entries and an omission marker, which can extend it beyond that target. The largest recorded complete orchestrator input is 10,510 tokens.

VideoLoop reduces orchestrator input by 48.0%, from 91.8K to 47.7K tokens per question compared with dual loop. We will document these input controls in Appendix H.

Reviewer_Xsq8OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies semantic thrashing in long-video agents: as append-only working memory accumulates observations, key evidence becomes increasingly difficult to access within a bounded context. VideoLoop’s core design combines an outer video-exploration agent with a persistent sandbox filesystem and an inner orchestrator that retrieves and rewrites bounded working memory after each step; the paper motivates this design with a formal analysis of append-only memory. Across three benchmarks and four LVLM backbones, together with memory ablations, context-only retrievability, and token-usage analyses, the full system shows consistent gains over native models and the append-only baseline. Overall, this is a clearly scoped system design with promising results, but the paper does not yet identify an independent mechanism beyond this composition or establish why this particular dual-loop structure is necessary relative to alternative memory managers. The contribution is therefore currently more convincing as an engineering integration than as a closed algorithmic advance.

Critical strengths
  1. VideoLoop has a clear system boundary: the outer agent explores the video and preserves historical artifacts, while the inner orchestrator rewrites working memory under a fixed budget. Compared with simple truncation or a sliding window, this addresses both premature loss of raw evidence and repeatedly carrying the full history in context (Sections 3.2–3.4, Figure 2).
  2. The paper reports more than final accuracy, including memory ablations, context-only retrievability, token usage, and multiple backbones. With Gemini 3 Flash, full VideoLoop improves from 81.9 for append-only to 85.8 while increasing total tokens by only about 0.5%; these results support a role for the memory design under the authors’ controlled setup.
Critical weaknesses
  1. The central contribution is not yet distinguished from an engineering composition. VideoLoop combines an outer video-exploration agent, a persistent artifact store, and an inner working-memory orchestrator; the results show that this composed system outperforms the append-only baseline, but do not isolate an independent new memory mechanism or explain why this particular coupling is necessary rather than attaching an existing retrieval, compression, backtracking, or hierarchical memory manager to the same outer agent. The current evidence therefore supports the practical value of the integration more clearly than a distinct and necessary algorithmic contribution.
  2. The token-efficiency claim does not directly establish practical inference efficiency. Table 3 reports only aggregate token volume, but the inner orchestrator rewrites M_t after every outer step while the outer context C_t={q,M_t,Ω_t} changes with it, potentially reducing prompt reuse and KV-cache reuse. The paper reports no cache-read and cache-write tokens, cache hit rate, latency, or provider cost, so the reported +0.5% token overhead cannot establish approximately constant inference cost.
  3. The formal and empirical links to semantic thrashing remain incomplete. Equations (3)–(8) rely on an unobserved target-evidence set K_q, while Figure 4’s blind judge measures answerability from frozen context rather than evidence correctness or faithfulness. The results therefore show improved context sufficiency, but do not directly validate the proposed failure mode or its mitigation.
Questions for authors

Can the authors report cache hit rate, cache-read and cache-write tokens, latency, and actual provider cost for append-only, dual-loop, and full VideoLoop under the same provider, caching policy, and runtime settings, to determine whether the +0.5% total-token overhead corresponds to approximately constant inference cost?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Xsq8 (Part 1/3: Q1 Contribution and Mem0)

Q1: What does VideoLoop contribute beyond combining existing components, and what supports its dual loop coupling?

A1: VideoLoop contributes a mechanism that couples raw observation retrieval with working memory rewriting. In long videos, a later clue often reveals which detail from an earlier observation matters. Having a separate inner orchestrator revisit original artifacts and rewrite working memory lets earlier details reenter reasoning while keeping the outer agent focused on video exploration.

To test attaching an existing memory manager to the same outer agent, as you suggest, we have compared VideoLoop directly with Mem0 [4]. VideoLoop achieves 85.8% accuracy versus 84.8% for Mem0. It also uses 19.2% fewer tokens and costs on average 16.2% less per question.

We keep the policy, tools, pinned transcript, eight recent turns, memory budget, and step limit the same. After each step, Mem0 extracts facts from the policy reasoning and tool result into a vector store. Before the next policy call, it retrieves facts using the question and latest reasoning and places them in the memory slot. Its memory calls use the same model as VideoLoop's orchestrator. Both methods run on the same questions.

MetricVideoLoopMem0
Accuracy85.8%84.8%
Main agent input511.3K626.1K
Memory manager input47.7K71.9K
Total tokens618.2K765.2K
Average cost per question$0.320$0.382

Because Mem0 extracts facts before later clues appear, missed details force the main agent to revisit the video, increasing main agent input from 511.3K to 626.1K tokens.

To clarify the design difference you ask about, the table compares who updates memory and what each update changes.

MethodMemory operationVideoLoop's design difference
MemGPT [1]The main LLM retrieves external memories and edits its own context through function callsA separate orchestrator handles retrieval and rewriting after each tool step that does not end the task, before control returns to the video policy
A Mem [2]New notes trigger links and updates to existing note attributes. Relevant notes are retrieved for a queryThe update target is the current task's working memory, rewritten using new observations and retrieved original evidence
VideoARM [3]Tool outputs enter Result Memory in temporal order. Working Memory records reasoning traces and goals, allowing tool context to be cleared after each callDuring memory maintenance, the orchestrator retrieves archived observations and integrates them into the rewritten working memory used for the next decision

We also test the roles of rewriting and retrieval separately, keeping the outer agent fixed across 900 VideoMME long video questions.

ConfigurationMemory updateAccuracyLatency per questionAgent calls per question
Append onlyAppend interaction history81.9%308.9 s29.8
Dual loopRewrite working memory without archive retrieval83.3%450.8 s15.6
VideoLoopRewrite with retrieval from the observation archive85.8%237.4 s13.6

Rewriting adds 13 correct answers, and archive retrieval adds 22 more. Adding retrieval also reduces latency from 450.8 s to 237.4 s and agent calls from 15.6 to 13.6 per question. These results connect the design to its benefit: retrieving original observations improves the accuracy and efficiency of continued video exploration.

References

[1] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560, 2023.

[2] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A Mem: Agentic Memory for LLM Agents. arXiv:2502.12110, 2025.

[3] Yufei Yin, Qianke Meng, Minghao Chen, Jiajun Ding, Zhenwei Shao, and Zhou Yu. VideoARM: Agentic Reasoning over Hierarchical Memory for Long Form Video Understanding. arXiv:2512.12360, 2025.

[4] Prateek Chhikara, Dev Khant, Saket Aryan, Taranjeet Singh, and Deshraj Yadav. Mem0: Building Production Ready AI Agents with Scalable Long Term Memory. arXiv:2504.19413, 2025.

(Continued in Part 2/3)

AuthorsOpenReview ↗

Response to Reviewer Xsq8 (Part 2/3: Q2 Cache, Latency, and Cost)

Q2: Does per step memory rewriting reduce prompt or KV cache reuse?

A2: Yes. Rewriting lowers prefix cache reuse from 86.9% to 54.6% and nearly doubles output tokens, so the 0.5% total token difference does not imply constant monetary cost ($0.320 versus $0.155 per question), although total latency drops by 23.1%.

To assess practical efficiency, we report cache reuse alongside agent calls, total latency, and cost under the same Gemini 3 Flash API setup.

VariantInput tokens served from cacheAgent calls per questionLatency per questionAverage cost per question
Append only86.9%29.8308.9 s$0.155
Dual loop53.6%15.6450.8 s$0.324
VideoLoop54.6%13.6237.4 s$0.320

The cached input percentage is the number of input tokens served from cache divided by total input tokens. Within VideoLoop, this is 59.5% for agent calls and 2.0% for orchestrator calls.

VideoLoop reduces agent calls from 29.8 to 13.6 per question and completes each question 23.1% faster, with a 3.9 percentage point accuracy gain. A2.d breaks down the cost into input and output.

Q2.a: What are the cache read and cache write token counts for each memory variant with the same model and API setup?

A2.a: Cached input per question is 508.0K tokens for append only, 316.4K for dual loop, and 305.0K for VideoLoop. The provider does not report cache write token counts in these usage fields.

The table separates cached and uncached input. All methods use the same model and API setup.

VariantInput tokensCache read tokensCache write tokensUncached input
Append only584.6K508.0KNot reported76.6K
Dual loop590.0K316.4KNot reported273.6K
VideoLoop559.0K305.0KNot reported254.0K

For VideoLoop, the 559.0K input tokens consist of 305.0K cached tokens and 254.0K uncached tokens. We apply the corresponding price to each category. Implicit caching has no separate write charge in this calculation.

Q2.b: What is the cache hit rate for each memory variant with the same model and API setup?

A2.b: Cached tokens account for 86.9% of input for append only, 53.6% for dual loop, and 54.6% for VideoLoop. The table also separates agent calls from orchestrator calls.

VariantAgent callsOrchestrator callsOverall
Append only86.9%N/A86.9%
Dual loop63.5%0%53.6%
VideoLoop59.5%2.0%54.6%

We measure reuse as cached input tokens divided by total input tokens. For VideoLoop, this is 305.0K / 559.0K = 54.6%.

Q2.c: What is the latency of each memory variant with the same model and API setup?

A2.c: VideoLoop takes 237.4 s per question, compared with 308.9 s for append only and 450.8 s for dual loop. Its total latency is 23.1% lower than append only.

The total includes memory maintenance, tools, and the other recorded overhead. The table separates agent calls from orchestrator calls.

VariantAgent LLM callsOrchestrator LLM callsTools and otherEnd to end
Append only292.4 sN/A16.5 s308.9 s
Dual loop185.6 s182.1 s83.1 s450.8 s
VideoLoop182.0 s27.3 s28.1 s237.4 s

Q2.d: What is the actual provider cost of each memory variant with the same model and API setup?

A2.d: Average cost per question is $0.320 for VideoLoop, $0.155 for append only, and $0.324 for dual loop. We price cached input, uncached input, and output separately to explain the difference.

The reported Gemini 3 Flash prices per million tokens are $0.05 for cached input, $0.50 for uncached input, and $3.00 for output.

VariantInput costOutput costTotal / question900 questions
Append only$0.064$0.091$0.155$139
Dual loop$0.153$0.171$0.324$291
VideoLoop$0.142$0.178$0.320$288

For the Mem0 comparison in A1, the table breaks down tokens and cost for each component, priced the same way.

ComponentVideoLoopMem0
Main agent input511.3K626.1K
Main agent output50.1K56.3K
Memory manager input47.7K71.9K
Memory manager output9.1K10.9K
Total input559.0K698.0K
Total output59.2K67.2K
Total tokens618.2K765.2K
Cached input305.0K374.0K
Uncached input254.0K324.0K
Cached input percentage54.6%53.6%
Main agent cost$0.269$0.314
Memory manager cost$0.051$0.068
Cost per question$0.320$0.382
Cost for 900 questions$288$344

VideoLoop reaches 85.8% accuracy at an average of $0.320 per question, compared with Mem0's 84.8% at $0.382. We will revise Section 4 and Table 3 to separate token volume, latency, and API cost rather than implying constant serving cost from token counts alone, with pricing details in Appendix A.

(Continued in Part 3/3)

AuthorsOpenReview ↗

Response to Reviewer Xsq8 (Part 3/3: Q3 Retention and Faithfulness)

Q3: How do Equations (3) to (8) validate semantic thrashing when KqK_q is unobserved?

A3: Equations (3) to (8) define the evidence maintenance problem, and the 3,387 annotated atomic fact probes serve as an observable ground truth subset of KqK_q to measure step by step evidence loss directly.

The fact retention test and the final task answer test address different questions. Appendix E tests whether the final memory supports the task answer across all 225 Q4 questions, with answer options visible.

MethodCorrect task answersFinal snapshot accuracy
Append only137/22560.9%
VideoLoop180/22580.0%

VideoLoop improves final snapshot accuracy by 19.1 percentage points. A3.a reports the separate experiment with 3,387 fact questions and hidden options, which tracks retention throughout each run.

Q3.a: What evidence validates semantic thrashing mitigation beyond improved context answerability?

A3.a: VideoLoop retains 88.7% of previously recovered facts, compared with 65.9% for append only. Its recovery rate also stays stable later in the run.

To address your question about evidence beyond final answer accuracy, we track the same 3,387 fact probes across 300 videos and 900 questions. Both methods use Gemini 3 Flash and the same tools. Qwen3.5 Flash reads the saved memory after each step without seeing the probe answer options.

The Appendix F results are shown below. The first table separates finding a fact at least once from retaining it until the final step.

MethodProbes recovered at least onceFinal retention among probes recovered at least once
Append only76.7%65.9%
VideoLoop77.6%88.7%

The next table compares recovery later in the run, using the progress interval where append only reaches its peak and the final interval.

MethodChange in fact recovery rate95% confidence interval for the change
Append only55.0% to 48.1%, a decrease of 6.9 percentage pointsDecrease of 4.5 to 9.5 points
VideoLoopIncrease of 1.1 percentage points over the same progress binsNegative 0.4 to positive 2.4 points

The confidence intervals use 1,000 bootstrap samples, with videos as the sampling unit. The larger budget comparison also supports retention: VideoLoop's final evidence loss is 11.3%, compared with 25.1% for append only with a 60K input budget.

Both methods recover a similar percentage of facts at least once. VideoLoop's gain comes from keeping those facts accessible as reasoning continues.

Q3.b: How is retained evidence checked for correctness and faithfulness?

A3.b: We compare recovered answers with annotated facts under hidden options to check factual correctness. For source faithfulness, Appendix K traces the rewritten memory directly against the original video evidence and tool outputs.

The checks and source comparison are described below. Appendix F evaluates 3,387 fact probes with answer options hidden, so the reader must produce the exact fact and any hallucinated claim or distorted value fails recovery. VideoLoop retains 88.7% of previously recovered facts, compared with 65.9% for the original baseline and 74.9% for the 60K baseline.

We normalize answers and accepted alternative forms before matching. An LLM grader checks other nonempty answers. Empty answers and responses that say the answer is unknown count as incorrect.

During memory updates, the orchestrator can retrieve original observations from the archive and read them alongside the current memory. In the spring compression example, the rewritten memory preserves the physical relation shown at 300 s and the Python result of 214.64 mm.

ItemOriginal evidenceRetained in the rewritten memory
Physical relationThe video frame at 300 s shows the energy conservation relation between gravity and spring energyW(h + x) = 0.5 k x², linked to the 300 s frame
Computed compressionPython returns x = 214.64 mmx = 214.64 mm, attributed to the Python computation
AnswerThe annotated answer is option A, 214.64 mmOption A, 214.64 mm

Here, x is the maximum spring compression. The rewritten memory preserves the physical relation, attributes the numerical result to Python, and retains the corresponding answer.

Comments on the paper

1 comment

We sincerely thank the reviewers for their detailed and constructive comments. Reviewer 5mtM recognizes the retrieval and rewrite design and the evidence retention analysis. Reviewer xJf3 recognizes the controlled memory comparisons and the method's use without training. Reviewer Xsq8 values the clear system boundary and the evaluation beyond final accuracy.

In response to your comments, we have added experiments and clarified the implementation. The main results are summarized as:.

  • [Reviewer 5mtM] VideoLoop reduces final evidence loss to 11.3%, compared with 25.1% for append only with a 60K input budget. We added this larger budget comparison to test whether the retention advantage comes from input capacity. The 60K budget exceeds VideoLoop's largest recorded policy input of 58.4K.
  • [Reviewer 5mtM] We report the input limits and the rules for handling oversized content in both loops. The largest recorded complete inputs are 58.4K tokens for the outer loop and 10,510 tokens for the inner loop.
  • [Reviewer 5mtM] Across all 225 Q4 questions, VideoLoop achieves 80.0% final snapshot accuracy, compared with 60.9% for append only. The corrected calculation counts all runs and yields a 19.1-percentage-point advantage.
  • [Reviewer xJf3] VideoLoop improves accuracy by 3.9 percentage points and reduces total latency by 23.1% compared with append only. The average costs are $0.320 and $0.155 per question, respectively. The time and cost totals include policy, tool analysis, and memory maintenance calls.
  • [Reviewer xJf3] Every step update achieves 85.8% accuracy, compared with 83.9% for periodic updates and 84.8% for triggered updates. To test the alternatives you raise, we have compared all three schedules across 900 VideoMME long-video questions.
  • [Reviewer Xsq8] With the same outer agent, VideoLoop achieves 85.8% accuracy versus 84.8% for Mem0 [4]. This comparison tests an existing memory manager with the outer agent held fixed. VideoLoop uses 19.2% fewer tokens and, on average, costs 16.2% less per question.
  • [Reviewer Xsq8] We report cached input, uncached input, and output tokens to explain the API costs. The calculation uses the same prices across all methods and accounts for the effect of memory rewriting on cache reuse.
  • [Reviewer Xsq8] VideoLoop retains 88.7% of previously recovered facts in the evaluation with answer options hidden. We explain how the 3,387 fact questions measure retention and show the source comparison from Appendix K in the response.

These results show how retrieving original observations and rewriting working memory improve evidence retention and video reasoning accuracy. We thank the reviewers again for their thoughtful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.