- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VideoLoop addresses the difficulty of retaining usable evidence during long video-agent trajectories, described as semantic thrashing. An outer multimodal policy explores the video with tools and stores observations in a persistent filesystem. After each step, a separate inner orchestrator retrieves relevant artifacts and edits a six-section working memory with a target token budget. A set-based analysis motivates deletion and recovery of evidence. Evaluation covers VideoMME (long), VideoMMMU, and LongVideoBench (long), with positive reported gains across four policy backbones. The paper also compares memory configurations, measures context-only question answerability, and introduces fact-level retention probes. The strongest empirical contribution is the combined system's accuracy improvement together with a more direct evaluation of whether previously recovered facts remain usable.
Critical strengths
-
The retrieval-and-rewrite design has useful empirical support. Preserving raw artifacts while editing the policy's current memory provides a practical way to revisit evidence. In Table 2, the Gemini 3 Flash comparison reports 81.9% for append-only, 83.3% for rewriting without filesystem retrieval, and 85.8% for the full system. The Q4 improvement from 70.7% to 80.0% is particularly relevant to the intended difficult questions. These are single-run results, but the component comparison, three-benchmark results in Table 1, and four-backbone results in Figure 3 collectively support the utility of this system combination.
-
The additional retention study provides information beyond final task accuracy. Appendix F evaluates 3,387 probes covering 300 videos and 900 questions, uses a different model family as the reader, hides probe answer options, and reports video-cluster bootstrap intervals. Among probes recovered at least once by each method, the final kept rates are 88.7% for VideoLoop and 65.9% for append-only (Figure 6). Separating initial recovery from subsequent availability is consequential for understanding memory behavior. The manually constructed probe subset and the stated caveats also make this evidence more informative than the headline answerability curve alone.
Critical weaknesses
-
The retention comparison does not isolate the proposed semantic-thrashing mechanism. Appendix F limits the append-only query plus history to 32K tokens and trims excess history, whereas VideoLoop's 32K limit applies to working memory before adding the query and recent interaction window. A matched total input budget is therefore not established. The appendix also acknowledges that it cannot separate transcript removal from difficulty recovering evidence that remains in context. These results support better evidence availability under the tested configurations, but leave the attention-dilution explanation in Sections 3.1 and 4.4 unresolved. This distinction matters because diagnosing and mitigating that failure mode is a central scientific claim. The acknowledged truncation and budget scope need to be reflected in that claim.
-
The fixed-context guarantee is not established for complete loop inputs. Section 3.3 concludes that the outer context has a constant bound, but limiting the recent window to eight triplets does not bound their token lengths. Tool observations can contain a full transcript or variable-length code output (Section 3.2). The inner loop also reads the manifest, filesystem index, and retrieved files without a stated total input limit. Algorithm 1, step 15, labels the memory limit a target; Appendices H and J do not specify token validation and oversize handling. Consequently, history growth could be shifted into the orchestrator, which would undermine the central scalability argument. Existing enforcement rules and input-length logs could resolve this concern; otherwise, the guarantee should be restricted to the components actually bounded.
Questions for authors
-
Existing configuration and log clarification. What hard limits and oversize-handling rules apply to both loops' complete inputs, including recent observations, manifest summaries, the filesystem index, retrieved files, and images? Please use existing specifications and length logs to explain the fixed-bound claim and whether Appendix F used equal total policy/judge budgets.
-
Suggested limited analysis. Could a small, manually checked subset of existing Appendix F snapshots distinguish facts removed by trimming from facts still present but no longer recovered? This would be additional analysis of existing runs. If unavailable, can the recognized truncation confound be reflected in the mechanism claim?
-
Aggregation clarification. Under Appendices D/E's 225-question, unsmoothed binary final-snapshot protocol, no integer count yields 81.1%; 182/225 and 183/225 round to 80.9% and 81.3%. What numerator, denominator, or averaging rule was actually used? Please correct the value or protocol as appropriate. The neighboring counts would preserve nearly all of the reported advantage.
If my concerns are adequately addressed, I would be willing to raise my score.
3 comments
Response to Reviewer 5mtM (Part 2/3: Q2 and Q2.a Loop Limits)
Q2: What hard token limits apply to the complete outer loop input?
A2: The 32K token limit applies to working memory. The complete outer request is assembled from the components below and has no separate total token check. The table lists the actual limits and handling rules for each component.
| Input component | Budget or handling |
|---|---|
| Stored working memory | Budget: 32K tokens. Oversized memory is shortened by section after an update, followed by text truncation if needed |
| Recent interactions | Retain the most recent 8 turns, with character clipping of policy text |
| Requested transcript | Retain one transcript block outside working memory |
| System instructions, query, and tool definitions | Add during request assembly |
| Video frame analysis | analyze_frames splits requested frames into batches of at most 64. Each analysis request keeps the latest image message. Subsequent ordinary policy calls remove image blocks from multimodal user messages |
| Question images | Attach during initial request construction, without a separate image count cap |
The largest recorded complete policy input is 58.4K tokens. When frame analysis uses the agent API, its input count includes image tokens. We include these calls in agent usage.
Q2.a: What hard token limits apply to the complete inner loop input?
A2.a: The inner loop limits text length, frame counts, and tool rounds. The table specifies what the memory orchestrator can receive. The orchestrator is the model responsible for retrieving evidence and rewriting working memory.
| Component | Implementation rule |
|---|---|
| Filesystem context | Clip the assembled view to 16,000 characters |
| Step records | Target: 16,000 characters. Keep at least two records and a marker for omitted content, which can extend the text beyond this target |
| New tool observation | Clip to 16,000 characters |
| Attached policy reasoning | Clip to 4,000 characters |
| File retrieval | read_file defaults to 4,000 characters and permits up to 8,000. Returned tool text is clipped to 4,000 on insertion |
| Frame inspection | Each frame inspection samples at most 6 frames |
| Frame selection | Frame selection samples at most 30 candidates and selects one for working memory |
| Images in rewrite calls | Initially attach question images. Before subsequent tool rounds, remove image blocks from messages preceding the last two messages |
| Tool rounds | At most 5 per rewrite |
The largest recorded complete inner call contains 10,510 input tokens. This count includes instructions, previous memory, accumulated tool exchanges, and image tokens reported by the provider.
(Continued in Part 3/3)
Response to Reviewer 5mtM (Part 3/3: Q2.b, Q2.c, and Q3)
Q2.b: How are oversized inputs handled in each loop?
A2.b: Oversized working memory is shortened by section, then truncated if it remains over the 32K budget. Retrieved text and other inner inputs are clipped before use. The table gives the handling rules.
| Content | Handling |
|---|---|
| Oversized working memory | Keep the last 10 nonempty lines of Key Evidence, shorten Current Understanding to 3 sentences, then truncate excess text |
| Filesystem view | Keep the beginning of the text, up to 16,000 characters |
| Retrieved file text | Read 4,000 characters by default, up to 8,000. Clip the returned text to 4,000 characters when adding it to the inner conversation |
| Step records | Target: 16,000 characters. Keep at least two records and a marker for omitted content, which can extend the text beyond this target |
| Video frame analysis | Split requested frames into batches of at most 64 and retain the latest image message. Ordinary policy calls then remove image blocks from multimodal user messages |
| Inner visual calls | Each frame inspection samples at most 6 frames. Frame selection samples at most 30 candidates |
| Images in rewrite calls | Before later tool rounds, remove image blocks from messages preceding the last two messages |
| Transcript | Retain one transcript block outside working memory |
| 60K append only control | Remove older messages until the question and history fit within the input budget |
Q2.c: Can the input length logs establish a fixed context limit?
A2.c: No. The logs establish observed input sizes. The largest recorded inputs are 58.4K tokens for the outer loop and 10,510 for the inner loop. To examine how input size changes during a run, we group the calls by reasoning step.
Each cell reports the median and maximum input tokens per call.
| Steps | Outer, VideoLoop | Outer, append only | Inner, VideoLoop |
|---|---|---|---|
| 1 to 5 | 17.7K / 44.4K | 10.4K / 60.0K | 3.6K / 10.5K |
| 6 to 15 | 23.0K / 58.4K | 16.3K / 60.0K | 5.1K / 10.1K |
| 16 to 50 | N/A | 7.6K / 60.0K | N/A |
Across these logged runs, all VideoLoop trajectories finished within 15 steps under the shared 50 step limit. We will revise Section 3.3 and Algorithm 1 to restrict our formal bound claim to the explicitly bounded components and report measured complete request sizes.
Q3: How was 81.1% computed under the 225 question, binary final snapshot protocol?
A3: The original 81.1% was 180/222. We now count the three runs that ended with errors as incorrect, giving 180/225 = 80.0%. Thank you for identifying the denominator issue.
| Method and calculation | Correct answers | Questions counted | Accuracy |
|---|---|---|---|
| VideoLoop, original calculation | 180 | 222 | 81.1% |
| VideoLoop, corrected calculation | 180 | 225 | 80.0% |
| Append only | 137 | 225 | 60.9% |
Across all 225 questions, VideoLoop achieves 80.0% accuracy versus 60.9% for append only, a 19.1 percentage point advantage. We will update Appendices D and E with the full denominator and corrected score.
Response to Reviewer 5mtM (Part 1/3: Q1 Budgets and Retention)
A1: No. The original 32K budgets covered different prompt components. Append only limited the question and history together to 32K tokens. VideoLoop applied the 32K limit to working memory, with the other prompt components added separately.
To test whether more input capacity removes the retention advantage, we have repeated the experiment with a 60K append only budget. We have also repeated the Appendix F evaluation. The 60K budget exceeds VideoLoop's largest recorded complete policy input of 58.4K.
The table tracks fact questions, or probes, with annotated answers. It reports the percentage of previously recovered facts that the reader cannot recover at each stage. For the final step, each method's denominator is all probes recovered at least once. Final loss is therefore 100% minus final retention.
VideoLoop reduces final evidence loss to 11.3%, compared with 25.1% for the 60K baseline, a difference of 13.8 percentage points. Its advantage holds at every reported stage. We will add this comparison and the denominator definition to Section 4.4 and Appendix F.
A1.a: No. Each method provides its full available memory snapshot, so reader input lengths differ. A snapshot is the memory saved at a particular reasoning step. All methods use the same reading protocol, without extra truncation and within the provider's context limit.
A1 addresses the policy input budgets. Here, we evaluate the memory retained by each method. The reader tests whether it can recover facts from the memory that each method retains.
Across the 3,387 fact probes with answer options hidden, VideoLoop retains 88.7% of previously recovered facts. Append only retains 65.9% with a 32K budget and 74.9% with a 60K budget. A1.b breaks down the final outcomes.
A1.b: Partly. The check identifies cases where the answer text remains in the snapshot but the reader cannot recover the answer. If the text is absent, it can have been trimmed or rewritten in different words.
To distinguish these outcomes, we have searched each final snapshot for the annotated answer and its accepted alternative forms. The table separates successful recovery, failure with no text match, and failure with a text match.
Each column counts probes recovered at least once by that method. VideoLoop reduces unrecovered probes with no answer text match from 23.0% for the 60K baseline to 8.5%.
A1.c: Under bounded budgets, append only loses recoverable facts mainly through history trimming once raw observations hit the cap, alongside context dilution on retained text.
Even at a 60K budget exceeding VideoLoop's 58.4K maximum, append only hits the cap and loses 25.1% of recovered facts versus 11.3% for VideoLoop.
We will revise Sections 3.1 and 4.4 to scope our claim around joint trimming and dilution under bounded context, rather than attention dilution alone.
(Continued in Part 2/3)