acceptodds
Under review as a conference paper at ICLR 2027

VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding

Abstract

Long-video understanding places substantial demands on memory, as answering questions often requires retrieving information distributed across extended temporal spans. Existing approaches broadly follow two paradigms: query-driven exploration, which is sensitive to localization errors, and query-independent memory construction, which may omit question-specific details. We introduce **VideoTapestry**, a training-free multi-agent framework that adapts a preconstructed hierarchical video memory through coarse-to-fine, query-driven refinement. The preconstructed memory organizes video content into three levels, capturing global narrative context, event-level temporal structure, and fine-grained relational evidence, respectively. To support coarse-to-fine localization and observation, we assign a specialized agent to each level, keeping retrieval and refinement within a scale-specific context. Guided by the query, these agents revisit relevant video regions and enrich layer-wise memories with targeted multimodal observations. Their refinements are assembled according to the original hierarchy into a composite query-adaptive memory, preserving global context in a compact form while retaining fine-grained evidence along query-relevant branches for final reasoning. Compared with direct GPT-5.5 inference, VideoTapestry achieves absolute accuracy gains of 17.2%, 14.9%, 9.8%, and 7.0% on LVBench, LongVideoBench (Long), Video-MME (Long), and EgoSchema, respectively, achieving the state-of-the-art results among all competitors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-Video Understanding、Hierarchical Multi-Agent Reasoning、Query-Adaptive Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ze45OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VideoTapestry, a training-free multi-agent framework for long-video understanding. It employs three specialized agents to progressively retrieve and refine preconstructed hierarchical video memory using query-relevant visual and audio evidence. The resulting query-adaptive memory is used for final question answering, achieving strong performance across four long-video benchmarks.

Critical strengths
  1. Important and timely problem: The paper addresses the important and timely challenge of long-video understanding, particularly the difficulty of retrieving fine-grained evidence while preserving global context.
  2. Intuitive and well-structured framework: The coarse-to-fine multi-agent design effectively combines global video context with fine-grained, query-specific evidence.
Critical weaknesses
  1. Limited Fairness and Comparability of Baseline Evaluations: The GPT-5.5 Direct baseline uses only 50 uniformly sampled frames from the entire video, whereas VideoTapestry leverages preconstructed hierarchical memory, additional visual observations, and Whisper-based ASR. Consequently, VideoTapestry has access to substantially more information than the Direct baseline. In particular, the paper does not clarify whether ASR information is provided to the Direct baseline. Given that removing ASR reduces VideoTapestry's accuracy by 5.5 percentage points, differences in the available input information may account for a substantial portion of the reported improvements. Furthermore, the results of other direct VLMs are partly taken from prior studies with potentially different evaluation configurations, making direct comparisons less reliable. This concern is particularly important because the reported performance of GPT-5.5 Direct is unusually low compared to several other direct VLM baselines.

  2. Insufficient Evidence for the Effectiveness of the Core Memory Refinement Mechanism: Although the paper reports substantial improvements over direct inference, these gains reflect the combined effects of additional evidence acquisition, preconstructed memory, multi-agent collaboration, and memory refinement. In particular, structured memory refinement yields only a 1.5 percentage-point improvement over the Append-only Evidence baseline, which uses the same tools and agent structure. Therefore, it remains unclear to what extent the proposed query-adaptive memory refinement mechanism itself, rather than the additional evidence acquisition and other components, is responsible for the reported performance gains.

  3. Lack of Computational Cost and Efficiency Analysis: The proposed framework requires not only offline memory construction but also multiple agent-based planning, observation, memory update, and ASR calls during query-time inference. However, the paper does not provide a quantitative analysis of offline construction costs, per-question token consumption, inference latency, or API/tool-call costs. Without such an analysis, it is difficult to assess whether the proposed approach offers practical advantages over existing methods when the additional computational overhead is taken into account.

Questions for authors

Q1. Could the authors clarify whether Whisper-based ASR or equivalent audio evidence was provided to the GPT-5.5 Direct baseline? It would be helpful to include a stronger Direct baseline evaluated under a comparable visual and audio evidence budget to better isolate the benefits of the proposed framework.

Q2. Could the authors provide a quantitative analysis of VideoTapestry's offline memory construction cost, average per-question token consumption, number of tool calls, inference latency, and API cost? Additionally, a comparison with existing methods under a matched inference budget would help clarify the practical efficiency and cost-effectiveness of the proposed approach.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ze45 (Part 1/2)

Thank you for the careful reading of our baselines, and for recognizing the importance of the problem and the coarse-to-fine design that combines global context with query-specific evidence. Revisions are marked in blue in the updated PDF.

W1 / Q1. Is the GPT-5.5 Direct baseline given comparable visual and audio input?{\color{#2563eb}\text{W1 / Q1. Is the GPT-5.5 Direct baseline given comparable visual and audio input?}}

"...the paper does not clarify whether ASR information is provided to the Direct baseline." (W1; Q1)

What Direct receives. The GPT-5.5 Direct baseline receives 50 uniformly sampled frames and no Whisper ASR. The 50-frame setting is the per-request image limit of the API gateway we used (Reproducibility Statement); the same limit bounds every Visual Observation call in VideoTapestry. Subtitle conditions are matched: on LongVideoBench both Direct and VideoTapestry use the official subtitles, and on Video-MME both are evaluated in the without-subtitles setting.

What VideoTapestry receives. It receives no additional annotation. ASR is a tool that the Overview Agent calls on windows it selects (at most three windows of 600–1200 s per question); the full video is not transcribed in advance. Audio is part of the source video, and deciding where to listen is part of the method. We agree, however, that this is an information difference relative to a frames-only Direct baseline, so the Direct comparison alone cannot attribute the gain. The following comparisons control for it.

  • Agentic baselines with the same backbone (Table 1). MemDreamer (same GPT-5.5, same preconstructed memory) and VideoXAgent (GPT-5.5, new blue row) are themselves multi-step, tool-using systems. VideoTapestry exceeds MemDreamer by 4.4–8.2 pp and VideoXAgent by 0.4–5.8 pp.
  • Without ASR (Table 3a). With ASR removed, VideoTapestry reaches 81.5% on the 200-question subset, against 78.0% for GPT-5.5 Direct on the same questions (Table 5). A gain remains with visual observation only.
  • More frames for Direct (Table 5). Qwen3.6-27B Direct receives 396 frames and Qwen3.7-Plus Direct 196; VideoTapestry still improves on them by 9.5 and 8.0 pp.
  • A stronger Direct baseline with full visual and audio input (new, Sec. 4.5.1, Table 4). Gemini-3.1-Pro processes the complete video natively.
Method (Gemini-3.1-Pro)LVBenchVideo-MME Long (w/o)
Direct78.280.3
MemDreamer (reproduced)79.781.5
VideoTapestry83.085.4

The Direct numbers are the reported Gemini-3.1-Pro results under its native protocol, in which the complete video is uploaded, so the model has access to both the frames and the audio track. Against this Direct baseline, which is limited neither to 50 frames nor to vision only, VideoTapestry remains 4.8 and 5.1 pp higher.

On the low GPT-5.5 Direct score. We agree that 50 frames is a tight budget for hour-long videos and that it explains part of the gap to Direct VLMs evaluated with native video input or more frames. For this reason we do not base our claims on the Direct comparison alone and added the Gemini-3.1-Pro experiment. Direct VLM numbers taken from prior reports are listed for reference and may use different settings, as the Reproducibility Statement notes.

AuthorsOpenReview ↗

Response to Reviewer Ze45 (Part 2/2)

W2. How much of the gain comes from memory refinement itself?{\color{#2563eb}\text{W2. How much of the gain comes from memory refinement itself?}}

"...it remains unclear to what extent the proposed query-adaptive memory refinement mechanism itself, rather than the additional evidence acquisition and other components, is responsible for the reported performance gains." (W2)

Table 3 separates the factors you list. All rows use GPT-5.5, the same 200 questions, and the same preconstructed memory.

Factor isolatedComparisonEffect
Preconstructed memory + hierarchical retrieval aloneStatic Memory Only (79.5) vs. Direct (78.0, Table 5)+1.5 pp
Query-time exploration and memory refinement (Table 3b)VideoTapestry (87.0) vs. Static Memory Only (79.5)+7.5 pp
Memory configuration under identical evidence (Table 3b)VideoTapestry (87.0) vs. Refined Super Only (85.0) / Refined Paths Only (80.5)+2.0 / +6.5 pp
Scale-aligned multi-agent collaborationVideoTapestry (87.0) vs. Single Agent (82.5)+4.5 pp
Visual observation / ASR observationVideoTapestry (87.0) vs. w/o Visual (84.0) / w/o ASR (81.5)+3.0 / +5.5 pp
Evidence placement policy (Table 3a)memory nodes (87.0) vs. flat log (85.5)1.5 pp

On the 1.5 pp figure. Append-only Evidence is not a baseline without memory refinement. It keeps the same agents, tools, and query-conditioned observations and changes only the placement policy for new evidence: a flat log instead of the corresponding memory nodes. The small gap shows that the framework is stable under either placement policy; over five repeated runs the standard deviation of accuracy is about 0.8 pp.

Separating refinement from evidence acquisition. The relevant comparison is Table 3(b). Its refined variants share the same retrieval results, observations, and query-time trace (Appendix C), so the acquired evidence is identical and only the memory given to the Answer Agent differs. Accuracy still ranges from 80.5% to 87.0%, and the composite refined memory improves over static memory by 7.5 pp. Keeping the global scaffold together with the refined branches is better than either alone. We added a paragraph in Sec. 4.4 defining each variant.

W3 / Q2. Computational cost and practical efficiency{\color{#2563eb}\text{W3 / Q2. Computational cost and practical efficiency}}

"...the paper does not provide a quantitative analysis of offline construction costs, per-question token consumption, inference latency, or API/tool-call costs." (W3; Q2)

We added Appendix E (Table 7), which covers each quantity you list. Setting: 10 LVBench videos (30.4–66.6 min, mean 46.2 min), 30 questions; Gemini-3.1-Pro for offline construction and GPT-5.5 online for all methods.

MethodOffline time (min/video)Offline API callsOffline tokens (M)Online latency (s/question)Online API callsOnline tokens (M)
MemDreamer75.85101.601.5983.8012.270.05
VideoLucy24.0448.000.29713.8028.970.35
Deep Video Discovery109.59287.900.79179.2810.030.17
VideoTapestry75.85101.601.59305.5218.700.27
  • Offline memory construction: 75.85 min, 101.6 API calls, and 1.59M tokens per video. It is shared with MemDreamer, performed once per video, and reused for all questions.
  • Per-question tokens and calls: 0.27M tokens and 18.70 API calls (model and tool calls together).
  • Latency: 305.52 s per question in our API environment.
  • API cost: at official API prices, $7.17 per video for offline memory construction with Gemini-3.1-Pro and $1.67 per question for online inference with GPT-5.5.

Online, VideoTapestry costs more than MemDreamer and Deep Video Discovery and less than VideoLucy on all three measures. We did not run an accuracy comparison at a matched inference budget. What we can report is that, with the same GPT-5.5 backbone and the same offline memory, the additional online computation over MemDreamer accompanies a gain from 73.3% to 81.5% on LVBench. We do not claim an efficiency advantage, and the limitations section (Appendix F) states this. Budget-aware observation is the natural next step.

Reviewer_Ze45OpenReview ↗

Follow-up

Thank you for the response. The cost analysis (Appendix E) resolves W3. W1 and W2 remain, and I keep my score.

1. The effect of memory refinement is not established (W2). Table 3(a), titled "Query-time memory refinement," defines Update as "structured evidence integration," and Append-only Evidence is the only row without it, at −1.5 pp (3/200 questions). Calling Append-only "not a baseline without memory refinement" holds only if a flat evidence log also counts as refinement, which makes refinement indistinguishable from evidence acquisition. Table 3(b) does not isolate refinement either. Its 80.5–87.0% range compares only refined variants, and the 7.5 pp gain over Static Memory Only "measures the combined effect of query-time observation and memory refinement" (Appendix C). The revised Sec. 4.4 itself concludes that the benefit "is not sensitive to where the acquired evidence is placed," yet the title and first contribution still present refinement as the core contribution.

2. Direct is still not matched in input (W1/Q1). The 50-frame limit is attributed to the API, yet the Overview Agent packs 120 frames into 20 grid images under the same limit. On Video-MME (w/o subtitles), VideoTapestry transcribes up to 60 minutes of audio per question for 30–60 minute videos (Algorithm 2), and removing ASR alone costs 5.5 pp. The baseline requested in Q1 was not provided: GPT-5.5 Direct with the full Whisper transcript and grid-packed frames.

3. The memory builder is not controlled. Appendix E states that the memory is built with Gemini-3.1-Pro, whose Direct LVBench score is 78.2, against 64.3 for GPT-5.5. If this memory is used in Tables 1, 3, and 5, the "Controlled Comparison" and the per-backbone gains include Gemini-3.1-Pro perception. Please state which model builds the memory for each result.

AuthorsOpenReview ↗

Response to Reviewer Ze45 (follow-up)

Thank you for the follow-up and for confirming that the cost analysis resolves W3. We respond to the three points in order and state plainly where we agree.

1. Refinement versus evidence acquisition{\color{#2563eb}\text{1. Refinement versus evidence acquisition}}

We agree that, in the terminology of Table 3(a), Append-only Evidence is the variant without structured integration; "placement policy" in our previous reply referred to the same distinction. In the paper, however, memory refinement denotes more than this storage choice. It is the query-adaptive memory strategy of our first contribution: compressing unrelated SuperEvents, enriching relevant nodes with new observations, and assembling refined branches under the preserved global scaffold (Sec. 3). The ablations support each part:

  • Integration into selected nodes versus a flat log: +1.5 pp, about twice the run-to-run standard deviation (0.8 pp over five runs). It is consistent and the smallest part.
  • Preserving the refined global scaffold alongside the refined branches: with the same query-time trace, removing it lowers accuracy from 87.0% to 80.5% (Table 3b).
  • The strategy as a whole versus the static memory: 87.0% versus 79.5%, with a shorter final memory (4,417 vs. 4,874 tokens). As Appendix C notes, this comparison also includes query-time observation.

Refinement is therefore distinguishable from evidence acquisition. Table 3(b) holds acquisition fixed, since all variants share the same retrieval results, observations, and trace, and memory organization alone still moves accuracy by 6.5 pp. The largest effect comes from keeping the refined global scaffold together with the refined branches, which is what our first contribution states.

2. Input matching for the Direct baseline{\color{#2563eb}\text{2. Input matching for the Direct baseline}}

The baseline you requested. We ran GPT-5.5 Direct with grid-packed frames and the full transcript, on the same test sets as Table 1:

  • Frames: 196 frames sampled uniformly over the whole video, packed into 49 2×2 grids (aspect ratio preserved, each frame labeled with its index and timestamp), within the same 50-image upload limit.
  • Transcript: all official subtitles of the video with timestamps, given as text, neither filtered by the sampled frames nor truncated.

On LongVideoBench Long, subtitles are part of the official evaluation protocol, which we follow: Direct receives the complete subtitles, while VideoTapestry is deliberately restricted to the subtitles of the windows it selects, without running Whisper. On Video-MME Long, which Table 1 reports without subtitles, we deliberately gave Direct the complete subtitles to provide the stronger Direct baseline suggested in Q1, while VideoTapestry stays without subtitles and uses Whisper only on its selected windows.

Method (GPT-5.5)LongVideoBench LongVideo-MME Long
Direct, 50 frames (Table 1)64.074.8
Direct, 196 grid-packed frames + full subtitles75.282.6
VideoTapestry (Table 1)78.984.6

The stronger baseline narrows the gap, and VideoTapestry remains 3.7 and 2.0 pp higher. The input comparison is no longer tilted toward VideoTapestry:

  • For whole-video coverage, Direct sees 196 frames, while the Overview Agent samples at most 120; Skim and Focus add frames only inside the segments selected for the question.
  • VideoTapestry's frames never carry burned-in subtitles or transcript text.
  • The tool is typically called once per question. From the recorded time ranges with GPT-5.5, the selected windows cover on average 16.0 minutes per question on Video-MME Long (900/900 questions; 30–60-minute videos) and 10.6 minutes on LongVideoBench Long (564/564 questions; 15–60-minute videos), well below the 60-minute upper bound set by three 600–1200 s windows.

We will add this baseline to the final version.

3. Which model builds the memory{\color{#2563eb}\text{3. Which model builds the memory}}

The hierarchical memory is built once per video with Gemini-3.1-Pro, following the MemDreamer procedure, and the same memory is used for every VideoTapestry and MemDreamer result in Tables 1–5, including the GPT and Qwen backbones. The memory is a shared, query-independent input, used in the same way by MemDreamer. Two comparisons hold it fixed and show that the gains come mainly from the query-time framework:

  • MemDreamer with GPT-5.5 (Table 1) uses the same Gemini-built memory and is 4.4–8.2 pp below VideoTapestry.
  • Static Memory Only (Table 3b) uses the same memory without query-time observation and reaches 79.5%, close to the 78.0% of GPT-5.5 Direct on the same 200 questions, whereas VideoTapestry reaches 87.0%. Of the 9.0 pp gain over Direct, 7.5 pp comes from query-time processing on top of the same memory.

We will state the memory builder in the experimental setup of the final version.

Reviewer_NqkrOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles the problem of long-video understanding and proposes VideoTapestry, a training-free multi-agent framework. It refines a preconstructed hierarchical memory (SuperEvents, MacroEvents, Subgraphs) via coarse-to-fine, query-driven agents (Overview, Skim, Focus) following a Plan-Observe-Update-Decide protocol. Refined layers are assembled into a composite query-adaptive memory for a final Answer Agent. Experiments are conducted on the LVBench, LongVideoBench, Video-MME, and EgoSchema datasets.

Critical strengths
  1. The motivation is clear and its writing is easy to follow, and the method section is well structured.
  2. The ablation studies demonstrate the effectiveness of the main components.
Critical weaknesses
  1. The core components, including hierarchical video memory, multi-agent decomposition, and the Plan–Observe–Update–Decide loop, are all adapted from prior work (e.g., the memory construction is explicitly taken from MemDreamer without modification), so the contribution reads more as a integration of existing ideas than a fundamentally new method.
  2. Despite being a training-free multi-agent framework with many query-time model and tool calls, the paper reports no token usage, call counts, or latency, so the substantial accuracy gains cannot be weighed against their computational price.
  3. Different methods in the comparison tables rely on very different base models. As the table in the paper of MemDreamer shows, MemDreamer with Gemini-3.1-Pro reaches much higher scores than VideoTapestry, so additional comparison under an identical backbone is needed before claiming superiority.
  4. The paper only presents successful qualitative trajectories and never analyzes failure cases, leaving unclear when and why the framework breaks down.
Questions for authors

Please refer to the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Nqkr (Part 1/2)

Thank you for the review, and for the positive comments on the motivation, the structure of the method section, and the ablation studies. We number the four weaknesses W1–W4 in the order they appear. Revisions are marked in blue in the updated PDF.

W1. Novelty of the core components{\color{#2563eb}\text{W1. Novelty of the core components}}

"...the contribution reads more as a integration of existing ideas than a fundamentally new method." (first weakness)

We agree that hierarchical video memory, multi-agent decomposition, and a Plan–Observe–Update–Decide loop each have precedents, and we do not claim them individually. The two works address different questions. MemDreamer is primarily about how to construct a hierarchical memory offline. VideoTapestry takes that memory as given and is about how to use it at query time: how to retrieve from it coarse-to-fine and refine it for the question at hand. The memory is built following the MemDreamer procedure on purpose: holding the offline memory fixed lets us study the query-time mechanism in isolation.

What is new is how the two are coupled at query time:

  • The preconstructed memory is used as a prior for where to look, not only as text to retrieve. Each agent plans observations of the source video from the memory at its own level.
  • Evidence gathered at one level changes what happens next: it refines that level's memory, determines which children are exposed to the next agent, and sets the inspection objective handed down (Algorithms 1–2, Appendix B.1).
  • The final context keeps a compressed scaffold of the whole video together with refined branches, instead of only the retrieved fragments.

Two results isolate this mechanism from the borrowed components:

  • Same memory, same backbone, different query-time mechanism (Table 1). MemDreamer reproduced with GPT-5.5 and given the same preconstructed memory scores 4.4–8.2 pp below VideoTapestry on all five columns.
  • Same memory, with and without query-time exploration (Table 3). Static Memory Only, which retrieves hierarchically but performs no observation or update, reaches 79.5%; the full method reaches 87.0%. Replacing the three scale-aligned agents with a single controller under the same call budget costs 4.5 pp.

W2. Computational cost and efficiency{\color{#2563eb}\text{W2. Computational cost and efficiency}}

"...the paper reports no token usage, call counts, or latency, so the substantial accuracy gains cannot be weighed against their computational price." (second weakness)

We added Appendix E (Table 7), which reports the three quantities you list. Costs are measured on 10 LVBench videos (30.4–66.6 min, mean 46.2 min) with 30 questions, and all methods use GPT-5.5 at query time.

MethodLatency (s/question)API calls per questionTokens per question (M)
MemDreamer83.8012.270.05
VideoLucy713.8028.970.35
Deep Video Discovery179.2810.030.17
VideoTapestry305.5218.700.27

API calls include both model and tool calls. Per question, VideoTapestry uses fewer calls, fewer tokens, and less time than VideoLucy, and more than MemDreamer and Deep Video Discovery. The additional query-time computation goes to targeted re-observation of the video and memory refinement; with the same GPT-5.5 backbone and the same offline memory, it accompanies 81.5% on LVBench versus 73.3% for MemDreamer. We do not claim an efficiency advantage, and the limitations section says so. Latency comes from one API environment and depends on service load, so it should be read as indicative.

Reviewer_zT2eOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VideoTapestry, a training-free multi agent framework for long video QA that adapts a preconstructed hierarchical video memory through coarse-to-fine, query-driven refinement. The method employs scale-aligned Overview, Skim, and Focus agents to selectively recheck relevant video segments, update SuperEvent, MacroEvent, and Subgraph memories, and assemble a query-adaptive memory for response generation. Experiments on 4 benchmarks show improvements over direct VLM inference and prior agentic baselines, while ablation and cross-backbone studies support the benefits of targeted observation, hierarchical memory refinement, and scale-aligned collaboration.

Critical strengths

The overall design is well motivated and internally coherent, with the Overview, Skim, and Focus agents naturally aligned with the SuperEvent, MacroEvent, and Subgraph levels of the hierarchical memory. This organization provides a clear coarse-to-fine reasoning process for long-video understanding while preserving a compact global view of the video. In particular, retaining global context while allocating more detailed observation to query-relevant branches is a sensible design choice for balancing broad coverage and fine-grained evidence retrieval in long videos.

Critical weaknesses

W1. The empirical evidence is not sufficiently robust to support the strength of the paper’s claims.

Table 1 compares VideoTapestry against prior methods that use substantially different frontier backbones, including GPT-5, o3, Gemini, and Claude variants, which makes it difficult to separate the effect of the proposed framework from differences in backbone capability. The same-backbone Direct comparison is useful for showing that the framework improves over direct inference, but it does not fully establish that VideoTapestry is methodologically superior to prior agentic systems under a controlled model setting.

This concern is more significant for the ablation results, which are conducted on a fixed 200-question subset of Video-MME Long without reported repeated runs, variance, or confidence intervals. For example, replacing structured memory updates with append-only evidence reduces accuracy from 87.0 to 85.5, which corresponds to only three additional correct answers on this subset. Given that the pipeline involves multiple GPT-5.5 based planning, observation, rewriting, and decision steps, it is difficult to determine whether such small differences reflect stable improvements or run-to-run variation. Repeated evaluations or statistical uncertainty estimates are therefore important for substantiating the claimed benefits of the key design choices.

W2. The paper does not convincingly establish that query-adaptive hierarchical refinement constitutes a sufficiently novel and necessary methodological contribution.

The main components of the system, including hierarchical video memory, coarse-to-fine retrieval, query-conditioned re-observation, and specialized agent roles, are closely related to ideas already explored in prior long-video memory and agentic reasoning systems. The main distinction appears to be the integration of newly observed evidence back into a preconstructed hierarchy, yet the corresponding structured-update ablation provides only a 1.5-point improvement over flat append-only evidence, while the Refined Super Only variant already reaches 85.0 compared with 87.0 for the full composite memory. This makes it unclear how much of the performance gain specifically depends on the proposed multi-level refinement mechanism rather than on the broader combination of retrieval, re-observation, ASR, and a strong frontier backbone.

There is also a methodological tension between the motivation and the actual inference procedure. VideoTapestry motivates its design partly by the brittleness of top-down methods to localization errors, but its own traversal still restricts each lower-level candidate set to the children of the branches selected at the previous level. In the evaluated configuration, each visited level performs only one refinement batch and one decision, with no repeated within-level refinement or explicit backtracking mechanism. Therefore, an incorrect Overview or Skim selection can still propagate through the hierarchy without an obvious recovery path. Stronger evidence that the proposed refinement mechanism can recover from imperfect localization, and that it provides clear gains beyond simpler hierarchical retrieval and re-observation pipelines, would be necessary to establish the significance of the methodological contribution.

Questions for authors
  • Could the authors report whether the Table 3 ablations were repeated across multiple runs and, if so, provide variance or confidence intervals, especially for the 1.5-point gap between Append-only Evidence and VideoTapestry?

  • Could the authors provide a more controlled comparison against prior agentic methods using the same or closely matched backbone, to clarify how much of the reported SOTA improvement comes from VideoTapestry rather than backbone differences?

  • Could the authors clarify what empirical evidence specifically demonstrates that structured hierarchical memory refinement is necessary beyond simply appending newly observed evidence, given the relatively small gap between these variants?

  • Could the authors evaluate or analyze failure cases where the Overview or Skim Agent selects an incorrect branch, and clarify whether the current inference procedure has any mechanism to recover from such localization errors?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zT2e (Part 1/3)

Thank you for the detailed review, and for noting that the alignment of the Overview, Skim, and Focus agents with the memory hierarchy is coherent and that keeping global context while refining query-relevant branches is a sensible design. We respond to W1 and W2 together with the related questions. Revisions are marked in blue in the updated PDF.

W1 / Q1. Repeated runs and uncertainty for the Table 3 ablations{\color{#2563eb}\text{W1 / Q1. Repeated runs and uncertainty for the Table 3 ablations}}

"...it is difficult to determine whether such small differences reflect stable improvements or run-to-run variation." (W1, second paragraph; Q1)

We repeated the ablation runs five times. The run-to-run variance of accuracy is 0.6 (pp)², i.e., a standard deviation of about 0.8 pp, so the 1.5 pp gap between Append-only Evidence and VideoTapestry is roughly twice the run-to-run standard deviation. The 200-question Video-MME Long subset follows the protocol of VideoARM, which keeps the ablations comparable with prior work.

On what this gap measures: Append-only Evidence is not a variant without memory refinement. It keeps the same hierarchical agents, tools, and query-conditioned observations, and changes only the placement policy for new evidence (a flat log instead of the corresponding memory nodes). We therefore read the small gap as stability across placement policies: both reach 85.5–87.0%, well above static memory at 79.5%. Our claims do not rest on this gap. The ablation of the memory itself is Table 3(b), where the differences are several times the run-to-run standard deviation: 79.5% for Static Memory Only versus 87.0% for the composite refined memory (+7.5 pp), and 80.5% for Refined Paths Only versus 87.0% (+6.5 pp).

W1 / Q2. Controlled comparison with prior agentic methods{\color{#2563eb}\text{W1 / Q2. Controlled comparison with prior agentic methods}}

"...it does not fully establish that VideoTapestry is methodologically superior to prior agentic systems under a controlled model setting." (W1, first paragraph; Q2)

We agree that rows of Table 1 obtained with different backbones cannot support such a claim. The revision provides three same-backbone comparisons with agentic systems.

(i) MemDreamer with GPT-5.5 and the same preconstructed memory (Table 1). We reproduced MemDreamer's retrieval pipeline with GPT-5.5 and gave it the same hierarchical memory as VideoTapestry, so the two differ only in the query-time mechanism.

(ii) VideoXAgent with GPT-5.5 (Table 1, new blue row). We also reproduced VideoXAgent with GPT-5.5 on all four benchmarks.

Method (GPT-5.5)LVBenchLongVideoBench ValLongVideoBench LongVideo-MME Long (w/o)EgoSchema
Direct64.369.864.074.876.0
MemDreamer (reproduced)73.374.173.977.178.6
VideoXAgent (reproduced)76.780.078.579.677.2
VideoTapestry81.580.478.984.683.0

VideoTapestry is ahead of MemDreamer by 4.4–8.2 pp on every column. Against VideoXAgent it is ahead by 4.8, 5.0, and 5.8 pp on LVBench, Video-MME Long, and EgoSchema, and by 0.4 pp on both LongVideoBench columns, where the two methods are close.

(iii) Gemini-3.1-Pro as a shared backbone (new, Sec. 4.5.1, Table 4). VideoTapestry reaches 83.0 on LVBench and 85.4 on Video-MME Long, versus 79.7 / 81.5 for MemDreamer and 78.2 / 80.3 for Direct.

Table 5 (Table 4 in the original submission) further reports gains of 7.0–9.5 pp over Direct for GPT-5.5, GPT-5, Qwen3.7-Plus, and Qwen3.6-27B.

AuthorsOpenReview ↗

Response to Reviewer zT2e (Part 2/3)

W2 / Q3. Novelty and necessity of structured hierarchical refinement{\color{#2563eb}\text{W2 / Q3. Novelty and necessity of structured hierarchical refinement}}

"This makes it unclear how much of the performance gain specifically depends on the proposed multi-level refinement mechanism rather than on the broader combination of retrieval, re-observation, ASR, and a strong frontier backbone." (W2, first paragraph; Q3)

On novelty. We agree that hierarchical memory, coarse-to-fine retrieval, re-observation, and specialized agents each have precedents. We do not claim them individually. What we study is their coupling at query time: the preconstructed memory decides where each agent looks in the source video, and what is observed at one level refines that level's memory, determines which children the next agent sees, and sets the inspection objective handed down (Algorithms 1–2, Appendix B.1).

On how much depends on refinement. The backbone, the offline memory, and the 200 questions are fixed across Table 3, so the differences below are not due to the backbone.

Comparison (Table 3)What changesEffect
(b) Static Memory Only (79.5) → VideoTapestry (87.0)static memory → query-refined composite memory+7.5 pp
(b) Refined Paths Only (80.5) → VideoTapestry (87.0)add the refined Super-level scaffold+6.5 pp
(b) Refined Super Only (85.0) → VideoTapestry (87.0)add refined Macro/Subgraph paths+2.0 pp
(a) Single Agent (82.5) → VideoTapestry (87.0)one controller → three scale-aligned agents, same call budget+4.5 pp
(a) Append-only Evidence (85.5) → VideoTapestry (87.0)placement policy for new evidence: flat log → memory nodes+1.5 pp

On the 1.5 pp row. This row is not the ablation of memory refinement. Append-only Evidence runs the same exploration with the same agents, tools, and query-conditioned observations; only the placement of new evidence differs. The small gap shows that the framework is stable under either placement policy. The memory ablation is Table 3(b), where refinement improves over static memory by 7.5 pp.

On Refined Super Only reaching 85.0. This variant is not a setting without refinement. Its Super-level memory has already been refined at query time by the Overview Agent: evidence-linked SuperEvents and their neighbors are enriched with new visual and ASR observations, and the remaining ones are compressed (Sec. 3.2.1). Its closest unrefined counterpart is Static Memory Only at 79.5%. The 85.0% therefore already reflects refinement at the Super level, and the refined Macro/Subgraph paths add a further 2.0 pp on top. Table 3(b) holds the exploration trace fixed, so these differences come only from which refined components reach the Answer Agent (Appendix C).

What we claim. (a) Query-time exploration guided by the preconstructed memory is the main source of improvement over static hierarchical retrieval. (b) The composite refined memory is better than static memory and, with the exploration trace fixed, than either refined component alone, by 2.0–7.5 pp, with a final memory that is also shorter than the unrefined one (4,417 vs. 4,874 tokens). (c) The framework is not sensitive to the evidence placement policy (85.5% vs. 87.0%).

AuthorsOpenReview ↗

Response to Reviewer zT2e (Part 3/3)

W2 / Q4. Wrong branch selection and recovery from localization errors{\color{#2563eb}\text{W2 / Q4. Wrong branch selection and recovery from localization errors}}

"...an incorrect Overview or Skim selection can still propagate through the hierarchy without an obvious recovery path." (W2, second paragraph; Q4)

Your description of the inference procedure is accurate. Each lower-level candidate set is restricted to the children of the selected branches, each visited level performs one refinement batch and one decision, and there is no backtracking after a branch is selected. We do not claim that VideoTapestry is immune to localization errors. The revision adds Appendix D (Table 6) to measure how often they are corrected and how often they propagate.

What differs from purely top-down search. Three things happen before a branch is committed. Locator outputs are treated as unverified temporal hints rather than decisions. The Overview Agent then observes the video (visual and local ASR), updates the Super-level memory, and only afterwards selects SuperEvents. It may select several of them. In addition, the final memory keeps the refined scaffold of all SuperEvents, so the Answer Agent still sees a compressed view of the whole video when a selected branch is wrong.

How often initial localization is corrected (all LVBench questions, ground-truth time annotations, 80% coverage criterion).

MetricCountRate
Recovery from insufficient initial hints850 / 1,19371.25%
Recovery beyond initially associated SuperEvents67 / 23528.51%
Multi-SuperEvent selection among recovered cases271 / 85031.88%

The first row should be read with care. Of the 850 recovered cases, 399 had empty initial hints and 384 had hints whose associated SuperEvent was already wide enough to cover the evidence. The stricter second row counts only cases where Overview had to select regions outside the initially associated SuperEvents; it succeeds in 28.51% of them.

Where errors occur. Of 287 incorrect LVBench predictions, 283 have usable temporal annotations (4 are excluded because their time references cannot be parsed into valid intervals). Each is assigned to the earliest stage at which coverage of the ground-truth interval is lost.

Earliest failure stageCountRate
Overview localization failure126 / 28344.52%
Skim localization failure40 / 28314.13%
Downstream evidence or reasoning failure117 / 28341.34%

About 59% of errors are therefore wrong-branch errors at Overview or Skim that propagate as you describe, and about 41% occur after the correct region was reached. Appendix D.3 walks through one case of each kind. In UID 126, Overview selects a later look-alike segment for a "first occurrence" question and the error propagates to the answer. In UID 251, localization is correct at every level, but the Answer Agent follows a coarse-level count (four) over the correct fine-grained count (five).

In short, correction is possible before commitment and happens in a measurable share of cases, while errors made at the commitment step are not recovered. We state this in Appendix D and in the limitations (Appendix F). Post-selection backtracking and cross-level conflict resolution are the two most direct improvements this analysis points to.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful and constructive feedback. We have uploaded a revised PDF in which the main changes are marked in blue. The additions are:

  1. Same-backbone comparison with a further agentic baseline (Table 1, blue row). We reproduced VideoXAgent with GPT-5.5 on all four benchmarks, alongside the existing GPT-5.5 reproduction of MemDreamer.
  2. Comparison under a long-context backbone (Sec. 4.5.1, Table 4). Direct, MemDreamer, and VideoTapestry are compared with Gemini-3.1-Pro on LVBench and Video-MME Long.
  3. Localization correction and failure analysis (Appendix D, Table 6). A retrospective analysis over all LVBench questions using ground-truth temporal annotations, a three-way breakdown of incorrect predictions, and two representative failure cases.
  4. Computational cost analysis (Appendix E, Table 7). Offline and online time, API calls, tokens, and API cost for VideoTapestry, MemDreamer, VideoLucy, and Deep Video Discovery under one configuration.
  5. Clarified ablation design (Sec. 4.4, blue paragraphs). We now state which question each ablation answers and how each variant is defined.

Two clarifications recur across reviews, so we state them once here.

What the contribution is. We do not claim a new hierarchical memory structure or a new agent loop. The contribution is the query-time mechanism: a preconstructed hierarchical memory guides where each agent observes, and the evidence obtained at one level conditions branch selection and inspection objectives at the next level. Relative to MemDreamer, which is primarily about how to build the hierarchical memory offline, VideoTapestry is about how to use that memory and refine it for each question.

How to read the 1.5 pp gap in Table 3(a). Append-only Evidence is not a variant without memory refinement. It keeps the same agents, tools, and query-conditioned observations as VideoTapestry and changes only the placement policy for newly observed evidence: appended as a flat log, or written into the corresponding memory nodes. The row therefore compares two placement policies. The framework performs well under both (85.5% and 87.0%, against 79.5% for static memory), so it is not sensitive to this choice.

The ablation of the memory itself is Table 3(b):

Memory given to the Answer AgentAcc.
Static Memory Only (no query-time refinement)79.5
Refined Paths Only80.5
Refined Super Only85.0
VideoTapestry (composite refined memory)87.0

The refined variants share the same retrieval results, observations, and query-time trace (Appendix C). With the evidence held fixed, the memory configuration alone moves accuracy between 80.5% and 87.0%, and the composite refined memory improves over static memory by 7.5 pp.

We sincerely thank all reviewers for your time and thoughtful feedback throughout the review process. Your constructive comments have helped us strengthen our manuscript and improve the clarity of its presentation.

As the discussion period draws to a close very soon, we would be grateful to hear whether our responses have addressed your concerns or whether any questions remain. We would be happy to provide any further clarification before the deadline.

Thank you again for your time and continued engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.