- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes VideoTapestry, a training-free multi-agent framework for long-video understanding. It employs three specialized agents to progressively retrieve and refine preconstructed hierarchical video memory using query-relevant visual and audio evidence. The resulting query-adaptive memory is used for final question answering, achieving strong performance across four long-video benchmarks.
Critical strengths
- Important and timely problem: The paper addresses the important and timely challenge of long-video understanding, particularly the difficulty of retrieving fine-grained evidence while preserving global context.
- Intuitive and well-structured framework: The coarse-to-fine multi-agent design effectively combines global video context with fine-grained, query-specific evidence.
Critical weaknesses
-
Limited Fairness and Comparability of Baseline Evaluations: The GPT-5.5 Direct baseline uses only 50 uniformly sampled frames from the entire video, whereas VideoTapestry leverages preconstructed hierarchical memory, additional visual observations, and Whisper-based ASR. Consequently, VideoTapestry has access to substantially more information than the Direct baseline. In particular, the paper does not clarify whether ASR information is provided to the Direct baseline. Given that removing ASR reduces VideoTapestry's accuracy by 5.5 percentage points, differences in the available input information may account for a substantial portion of the reported improvements. Furthermore, the results of other direct VLMs are partly taken from prior studies with potentially different evaluation configurations, making direct comparisons less reliable. This concern is particularly important because the reported performance of GPT-5.5 Direct is unusually low compared to several other direct VLM baselines.
-
Insufficient Evidence for the Effectiveness of the Core Memory Refinement Mechanism: Although the paper reports substantial improvements over direct inference, these gains reflect the combined effects of additional evidence acquisition, preconstructed memory, multi-agent collaboration, and memory refinement. In particular, structured memory refinement yields only a 1.5 percentage-point improvement over the Append-only Evidence baseline, which uses the same tools and agent structure. Therefore, it remains unclear to what extent the proposed query-adaptive memory refinement mechanism itself, rather than the additional evidence acquisition and other components, is responsible for the reported performance gains.
-
Lack of Computational Cost and Efficiency Analysis: The proposed framework requires not only offline memory construction but also multiple agent-based planning, observation, memory update, and ASR calls during query-time inference. However, the paper does not provide a quantitative analysis of offline construction costs, per-question token consumption, inference latency, or API/tool-call costs. Without such an analysis, it is difficult to assess whether the proposed approach offers practical advantages over existing methods when the additional computational overhead is taken into account.
Questions for authors
Q1. Could the authors clarify whether Whisper-based ASR or equivalent audio evidence was provided to the GPT-5.5 Direct baseline? It would be helpful to include a stronger Direct baseline evaluated under a comparable visual and audio evidence budget to better isolate the benefits of the proposed framework.
Q2. Could the authors provide a quantitative analysis of VideoTapestry's offline memory construction cost, average per-question token consumption, number of tool calls, inference latency, and API cost? Additionally, a comparison with existing methods under a matched inference budget would help clarify the practical efficiency and cost-effectiveness of the proposed approach.
4 comments
Response to Reviewer Ze45 (Part 2/2)
"...it remains unclear to what extent the proposed query-adaptive memory refinement mechanism itself, rather than the additional evidence acquisition and other components, is responsible for the reported performance gains." (W2)
Table 3 separates the factors you list. All rows use GPT-5.5, the same 200 questions, and the same preconstructed memory.
| Factor isolated | Comparison | Effect |
|---|---|---|
| Preconstructed memory + hierarchical retrieval alone | Static Memory Only (79.5) vs. Direct (78.0, Table 5) | +1.5 pp |
| Query-time exploration and memory refinement (Table 3b) | VideoTapestry (87.0) vs. Static Memory Only (79.5) | +7.5 pp |
| Memory configuration under identical evidence (Table 3b) | VideoTapestry (87.0) vs. Refined Super Only (85.0) / Refined Paths Only (80.5) | +2.0 / +6.5 pp |
| Scale-aligned multi-agent collaboration | VideoTapestry (87.0) vs. Single Agent (82.5) | +4.5 pp |
| Visual observation / ASR observation | VideoTapestry (87.0) vs. w/o Visual (84.0) / w/o ASR (81.5) | +3.0 / +5.5 pp |
| Evidence placement policy (Table 3a) | memory nodes (87.0) vs. flat log (85.5) | 1.5 pp |
On the 1.5 pp figure. Append-only Evidence is not a baseline without memory refinement. It keeps the same agents, tools, and query-conditioned observations and changes only the placement policy for new evidence: a flat log instead of the corresponding memory nodes. The small gap shows that the framework is stable under either placement policy; over five repeated runs the standard deviation of accuracy is about 0.8 pp.
Separating refinement from evidence acquisition. The relevant comparison is Table 3(b). Its refined variants share the same retrieval results, observations, and query-time trace (Appendix C), so the acquired evidence is identical and only the memory given to the Answer Agent differs. Accuracy still ranges from 80.5% to 87.0%, and the composite refined memory improves over static memory by 7.5 pp. Keeping the global scaffold together with the refined branches is better than either alone. We added a paragraph in Sec. 4.4 defining each variant.
"...the paper does not provide a quantitative analysis of offline construction costs, per-question token consumption, inference latency, or API/tool-call costs." (W3; Q2)
We added Appendix E (Table 7), which covers each quantity you list. Setting: 10 LVBench videos (30.4–66.6 min, mean 46.2 min), 30 questions; Gemini-3.1-Pro for offline construction and GPT-5.5 online for all methods.
| Method | Offline time (min/video) | Offline API calls | Offline tokens (M) | Online latency (s/question) | Online API calls | Online tokens (M) |
|---|---|---|---|---|---|---|
| MemDreamer | 75.85 | 101.60 | 1.59 | 83.80 | 12.27 | 0.05 |
| VideoLucy | 24.04 | 48.00 | 0.29 | 713.80 | 28.97 | 0.35 |
| Deep Video Discovery | 109.59 | 287.90 | 0.79 | 179.28 | 10.03 | 0.17 |
| VideoTapestry | 75.85 | 101.60 | 1.59 | 305.52 | 18.70 | 0.27 |
- Offline memory construction: 75.85 min, 101.6 API calls, and 1.59M tokens per video. It is shared with MemDreamer, performed once per video, and reused for all questions.
- Per-question tokens and calls: 0.27M tokens and 18.70 API calls (model and tool calls together).
- Latency: 305.52 s per question in our API environment.
- API cost: at official API prices, $7.17 per video for offline memory construction with Gemini-3.1-Pro and $1.67 per question for online inference with GPT-5.5.
Online, VideoTapestry costs more than MemDreamer and Deep Video Discovery and less than VideoLucy on all three measures. We did not run an accuracy comparison at a matched inference budget. What we can report is that, with the same GPT-5.5 backbone and the same offline memory, the additional online computation over MemDreamer accompanies a gain from 73.3% to 81.5% on LVBench. We do not claim an efficiency advantage, and the limitations section (Appendix F) states this. Budget-aware observation is the natural next step.
Follow-up
Thank you for the response. The cost analysis (Appendix E) resolves W3. W1 and W2 remain, and I keep my score.
1. The effect of memory refinement is not established (W2). Table 3(a), titled "Query-time memory refinement," defines Update as "structured evidence integration," and Append-only Evidence is the only row without it, at −1.5 pp (3/200 questions). Calling Append-only "not a baseline without memory refinement" holds only if a flat evidence log also counts as refinement, which makes refinement indistinguishable from evidence acquisition. Table 3(b) does not isolate refinement either. Its 80.5–87.0% range compares only refined variants, and the 7.5 pp gain over Static Memory Only "measures the combined effect of query-time observation and memory refinement" (Appendix C). The revised Sec. 4.4 itself concludes that the benefit "is not sensitive to where the acquired evidence is placed," yet the title and first contribution still present refinement as the core contribution.
2. Direct is still not matched in input (W1/Q1). The 50-frame limit is attributed to the API, yet the Overview Agent packs 120 frames into 20 grid images under the same limit. On Video-MME (w/o subtitles), VideoTapestry transcribes up to 60 minutes of audio per question for 30–60 minute videos (Algorithm 2), and removing ASR alone costs 5.5 pp. The baseline requested in Q1 was not provided: GPT-5.5 Direct with the full Whisper transcript and grid-packed frames.
3. The memory builder is not controlled. Appendix E states that the memory is built with Gemini-3.1-Pro, whose Direct LVBench score is 78.2, against 64.3 for GPT-5.5. If this memory is used in Tables 1, 3, and 5, the "Controlled Comparison" and the per-backbone gains include Gemini-3.1-Pro perception. Please state which model builds the memory for each result.
Response to Reviewer Ze45 (follow-up)
Thank you for the follow-up and for confirming that the cost analysis resolves W3. We respond to the three points in order and state plainly where we agree.
We agree that, in the terminology of Table 3(a), Append-only Evidence is the variant without structured integration; "placement policy" in our previous reply referred to the same distinction. In the paper, however, memory refinement denotes more than this storage choice. It is the query-adaptive memory strategy of our first contribution: compressing unrelated SuperEvents, enriching relevant nodes with new observations, and assembling refined branches under the preserved global scaffold (Sec. 3). The ablations support each part:
- Integration into selected nodes versus a flat log: +1.5 pp, about twice the run-to-run standard deviation (0.8 pp over five runs). It is consistent and the smallest part.
- Preserving the refined global scaffold alongside the refined branches: with the same query-time trace, removing it lowers accuracy from 87.0% to 80.5% (Table 3b).
- The strategy as a whole versus the static memory: 87.0% versus 79.5%, with a shorter final memory (4,417 vs. 4,874 tokens). As Appendix C notes, this comparison also includes query-time observation.
Refinement is therefore distinguishable from evidence acquisition. Table 3(b) holds acquisition fixed, since all variants share the same retrieval results, observations, and trace, and memory organization alone still moves accuracy by 6.5 pp. The largest effect comes from keeping the refined global scaffold together with the refined branches, which is what our first contribution states.
The baseline you requested. We ran GPT-5.5 Direct with grid-packed frames and the full transcript, on the same test sets as Table 1:
- Frames: 196 frames sampled uniformly over the whole video, packed into 49 2×2 grids (aspect ratio preserved, each frame labeled with its index and timestamp), within the same 50-image upload limit.
- Transcript: all official subtitles of the video with timestamps, given as text, neither filtered by the sampled frames nor truncated.
On LongVideoBench Long, subtitles are part of the official evaluation protocol, which we follow: Direct receives the complete subtitles, while VideoTapestry is deliberately restricted to the subtitles of the windows it selects, without running Whisper. On Video-MME Long, which Table 1 reports without subtitles, we deliberately gave Direct the complete subtitles to provide the stronger Direct baseline suggested in Q1, while VideoTapestry stays without subtitles and uses Whisper only on its selected windows.
| Method (GPT-5.5) | LongVideoBench Long | Video-MME Long |
|---|---|---|
| Direct, 50 frames (Table 1) | 64.0 | 74.8 |
| Direct, 196 grid-packed frames + full subtitles | 75.2 | 82.6 |
| VideoTapestry (Table 1) | 78.9 | 84.6 |
The stronger baseline narrows the gap, and VideoTapestry remains 3.7 and 2.0 pp higher. The input comparison is no longer tilted toward VideoTapestry:
- For whole-video coverage, Direct sees 196 frames, while the Overview Agent samples at most 120; Skim and Focus add frames only inside the segments selected for the question.
- VideoTapestry's frames never carry burned-in subtitles or transcript text.
- The tool is typically called once per question. From the recorded time ranges with GPT-5.5, the selected windows cover on average 16.0 minutes per question on Video-MME Long (900/900 questions; 30–60-minute videos) and 10.6 minutes on LongVideoBench Long (564/564 questions; 15–60-minute videos), well below the 60-minute upper bound set by three 600–1200 s windows.
We will add this baseline to the final version.
The hierarchical memory is built once per video with Gemini-3.1-Pro, following the MemDreamer procedure, and the same memory is used for every VideoTapestry and MemDreamer result in Tables 1–5, including the GPT and Qwen backbones. The memory is a shared, query-independent input, used in the same way by MemDreamer. Two comparisons hold it fixed and show that the gains come mainly from the query-time framework:
- MemDreamer with GPT-5.5 (Table 1) uses the same Gemini-built memory and is 4.4–8.2 pp below VideoTapestry.
- Static Memory Only (Table 3b) uses the same memory without query-time observation and reaches 79.5%, close to the 78.0% of GPT-5.5 Direct on the same 200 questions, whereas VideoTapestry reaches 87.0%. Of the 9.0 pp gain over Direct, 7.5 pp comes from query-time processing on top of the same memory.
We will state the memory builder in the experimental setup of the final version.
Response to Reviewer Ze45 (Part 1/2)
Thank you for the careful reading of our baselines, and for recognizing the importance of the problem and the coarse-to-fine design that combines global context with query-specific evidence. Revisions are marked in blue in the updated PDF.
W1 / Q1. Is the GPT-5.5 Direct baseline given comparable visual and audio input?
What Direct receives. The GPT-5.5 Direct baseline receives 50 uniformly sampled frames and no Whisper ASR. The 50-frame setting is the per-request image limit of the API gateway we used (Reproducibility Statement); the same limit bounds every Visual Observation call in VideoTapestry. Subtitle conditions are matched: on LongVideoBench both Direct and VideoTapestry use the official subtitles, and on Video-MME both are evaluated in the without-subtitles setting.
What VideoTapestry receives. It receives no additional annotation. ASR is a tool that the Overview Agent calls on windows it selects (at most three windows of 600–1200 s per question); the full video is not transcribed in advance. Audio is part of the source video, and deciding where to listen is part of the method. We agree, however, that this is an information difference relative to a frames-only Direct baseline, so the Direct comparison alone cannot attribute the gain. The following comparisons control for it.
The Direct numbers are the reported Gemini-3.1-Pro results under its native protocol, in which the complete video is uploaded, so the model has access to both the frames and the audio track. Against this Direct baseline, which is limited neither to 50 frames nor to vision only, VideoTapestry remains 4.8 and 5.1 pp higher.
On the low GPT-5.5 Direct score. We agree that 50 frames is a tight budget for hour-long videos and that it explains part of the gap to Direct VLMs evaluated with native video input or more frames. For this reason we do not base our claims on the Direct comparison alone and added the Gemini-3.1-Pro experiment. Direct VLM numbers taken from prior reports are listed for reference and may use different settings, as the Reproducibility Statement notes.