acceptodds
Withdrawn from ICLR 2027

GOPAgen: Codec-Aware Agentic Long-Video Understanding with Structured Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic long-video question answering often relies on sparse RGB sampling and caption retrieval. This reliance can lead to missed brief motion events and repeated processing of irrelevant intervals. We introduce GOPAgen, a framework that integrates codec-native Groups of Pictures (GOPs) and motion vectors into an agentic retrieval pipeline. GOPAgen constructs global textual memory and uses two-stage query-conditioned temporal selection. A global-sufficiency check precedes local evidence extraction. Caption and motion agents represent selected intervals as typed memory pages containing complementary appearance and motion evidence, temporal metadata, and provenance. A GOP-Tree organizes these pages for conditional retrieval when the global evidence is insufficient. GOPAgen achieves 65.3% accuracy on MotionBench Test and 78.7% on EgoSchema, exceeding the reported agentic baselines listed for these benchmarks. It also achieves 73.2% on LongVideoBench validation and 77.3% on MLVU, while remaining below the strongest listed baseline on LVBench and Video-MME Long. These results support codec-aware structured memory as an effective interface for selective long-video reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long video understanding, agent, MLLM

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5ZmHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary This paper proposes GOPAgen which brings codec GOPs and motion vectors into agentic long video question answering. The system first checks global-frame sufficiency then uses query-conditioned two-stage temporal selection to locate relevant intervals. A codec-aware Motion Agent and a Caption Agent build typed memory pages and a GOP-Tree organizes ranks and compresses the evidence for final reasoning. It reports competitive results on MotionBench EgoSchema LongVideoBench and MLVU with lower online latency and includes fairly complete ablations on motion vectors tree structure optical flow window selection latency and failure modes. Contributions It uses codec motion vectors rather than optical flow as a complementary motion cue and converts them into retrievable text evidence through a Motion Agent. It builds an agentic pipeline with two-stage temporal selection and global-sufficiency gating that stays competitive at lower online latency. It provides No MV Reference-frame only and Optical Flow ablations that support the role of motion vectors. Validation The direction is valuable but the evidence is not yet sufficient. The compared methods are outdated and the main tables center on GPT-4o and Gemini-1.5/2.0 which cannot support a state-of-the-art claim. The figures are rough and do not match the two-stage temporal selection in the text. Candidate counts do not align with the Appendix B.3 description the GOP-Tree efficiency attribution does not hold and misses from sparse global sampling cannot be recovered. Most leads are within 2 pp and no significance is reported. Overall it is a borderline paper that a substantive rebuttal could change.

Critical strengths
  1. The entry point is reasonable. The paper does not equate codec motion vectors with optical flow but treats them as encoder prediction cues that need RGB reference frames for semantic interpretation.
  2. The usefulness of motion vectors is backed by ablations. No MV drops 4.5 pp on MotionBench relative to Full GOPAgen and Reference-frame only is also lower.
  3. Experimental coverage is fairly complete including backbone swaps component removal optical-flow and codec comparisons parameter sensitivity retrieval diagnostics and failure classification.
  4. The limitations are stated with restraint acknowledging that fewer visual tokens do not scale proportionally with FLOPs or total cost and that the method relies on several models and heuristic search.
Critical weaknesses
  1. The compared methods are clearly outdated. The main tables still center on GPT-4o and Gemini-1.5/2.0 which were no longer frontier at submission time and cannot support a state-of-the-art claim. A comparison against strong baselines already published by the deadline is needed.
  2. The figures are rough and do not match the text. The concept and pipeline diagrams are unclear and Figure 2 shows Coarse Fine and Super-Fine levels while the text and Appendix B.3 implement only "two rounds with a fixed search depth". The figures should be reconciled with the text and should show the global-sufficiency gate branch.
  3. The core motivation is not closed and the prompt is self-contradictory. The motivation states that sparse sampling can "miss a short-lived event" yet the Motion Agent only processes already-selected intervals and global inspection uses at most 32 frames for a 60-minute video so a short action missed at the global stage fails the whole chain. Evidence recall should be reported by action duration. In addition the text says the Motion Agent outputs "a fine-grained motion description, not a separate answer" while the prompt asks to adopt its "reference answer" which the appendix admits does not exist as a field.
Questions for authors
  1. The comparison set lags the field. The newest listed methods are from 2025 with no 2026 work, and the closed-source baselines are especially dated (GPT-4o, Gemini-1.5/2.0), which is too weak to support the state-of-the-art claim. Please add comparisons against strong models already published by the deadline, e.g. Qwen 3.8 27B (open) and GPT-5.6 (closed).
  2. Please define the generated/scored/retained candidate counts at each stage and give one full retrieval trace, explaining why the 4.8 and 6.2 in Tables 9/10 are far below the ~24 implied by B.3. Also, under the same selection and gating, the Flat-memory variant is actually faster (63.7 vs 65.0s), so where does the GOP-Tree's independent efficiency gain come from?
  3. When a key short action is absent from the 32 global frames, how does the system select the right interval? Can you report coarse/fine/evidence recall by action duration? Also, the text says the Motion Agent only outputs a motion description, yet the prompt asks the system to adopt its "reference answer," which does not exist. How is this reconciled?
  4. How do Figure 2's three levels (Coarse/Fine/Super-Fine) map onto the "two rounds with a fixed search depth" in the text? Is Super-Fine just the Motion Agent's internal GOP granularity rather than a separate selection stage?
  5. As frontier models rapidly improve, both closed (e.g., 6astra) and open (e.g., Qwen3.8-27B) models offer ever stronger native long-context video understanding. Is the video reasoning task this paper targets still a real problem? More importantly, relative to simply plugging such strong models into a general-purpose base harness (e.g., Codex), where does your method's value and irreplaceability lie? Please provide a direct comparison against a "base harness + frontier model" setup.
Reviewer_AUv6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GOPAgen, a codec-aware framework for long-video question answering that integrates motion-vector representations with structured textual memory. The system constructs a global representation from sparse RGB observations and performs query-conditioned, coarse-to-fine temporal selection. When global evidence is deemed insufficient, appearance and motion agents generate interval-specific descriptions, which are stored as memory pages with temporal metadata and provenance. A GOP-Tree organizes these pages for retrieval and subsequent language-model reasoning. The principal claims concern improved preservation of motion-sensitive evidence and reduced processing costs. The evaluation includes multiple video-understanding benchmarks, component ablations, retrieval diagnostics, and resource measurements.

Critical strengths

The paper addresses a relevant limitation of sparse-frame video understanding: temporally localized motion evidence may be absent from the sampled RGB observations. Incorporating codec-derived motion information is a reasonable direction, and associating appearance and motion descriptions with their temporal provenance provides a potentially useful interface for evidence retrieval.

The reported results are competitive on several benchmarks, particularly MotionBench and EgoSchema. The supplementary evaluation also extends beyond aggregate accuracy to include component comparisons, codec configurations, retrieval behavior, and stage-wise resource accounting. These measurements provide useful evidence about the implemented system, although they do not fully substantiate the proposed explanations for its performance and efficiency.

Critical weaknesses

1.The evaluation does not rigorously validate the core advantage of the codec motion representation against budget-equivalent RGB alternatives, and the standalone motion-agent results lack clear comparability. The key claim that motion vectors recover details missed by sparse RGB sampling lacks a controlled comparison with denser RGB sampling under the same compute budget; existing ablations only confirm the motion branch is useful, not that it yields a better accuracy–cost trade-off. Temporal alignment between codec reference pictures and input RGB frames is insufficiently specified, weakening the claimed temporal detail recovery. In addition, Table 3 mixes this work’s multiple-choice evaluation protocol with externally sourced EMA results scored by GPT-3.5-turbo, without clearly distinguishing reproduced results from prior values, so the reported motion-agent gains are not well-supported.

2.Ablations entangle multiple factors, limiting reliable attribution of gains, and the claimed efficiency benefits cannot be clearly attributed to the codec design due to ambiguous end-to-end cost accounting. Key ablation tables modify multiple variables together and omit whether ablated variants are retrained, making it hard to separate true representation gains from input distribution shifts. Cross-benchmark analysis is descriptive rather than diagnostic and fails to quantify performance breakdowns between temporal selection errors and motion-description errors. The observed latency reduction is a system-level outcome confounded by different evidence pipelines and model call counts, lacking a controlled RGB-only baseline. Cost accounting also has ambiguities around transcoding overhead and memory page construction cost type, obscuring end-to-end efficiency claims for both first and amortized multi-query scenarios.

Questions for authors
  1. Motion representation. What is the exact reference picture associated with each motion record, and how does it relate to the I/IDR image supplied to the model? Is reference-picture identity retained? Please define the temporal offset and specify which timestamps remain represented after tokenization and subsampling, particularly those absent from the RGB input.

  2. Controlled performance comparison. With temporal selection, downstream reasoning, and model and training conditions controlled, how does sparse RGB plus motion vectors compare with denser uniform RGB sampling at comparable measured computational cost? Please also specify the distinction between “No motion-vector input” and “Reference-frame only,” and whether input-removal and input-substitution variants are retrained or adapted.

  3. End-to-end efficiency. Does Table 8 include the video transcoding required by Appendix B.2? Which memory-construction operations are reusable across questions, and which are query-dependent? How do first-query and amortized multi-query costs compare when all necessary preprocessing and online operations are included consistently for each method?

  4. Evaluation protocol. Do the MSVD-QA, MSRVTT-QA, and ActivityNet-QA results in Table 3 use generated-answer evaluation or multiple-choice matching? Which baseline results were reproduced, and were all methods evaluated using the same question format and scoring procedure?

Reviewer_XfMjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

For successful agentic long video understanding, the paper focuses on two challenges: 1) missing brief movements across sampled frames and 2) followed computation increments from repeatedly inspecting relevant intervals. To tackle these, the paper leverages GOP structures in compressed videos. Namely, the propposed agentic pipeline first sparsely inspects the video, and then skips local extraction when the global memory is determined to be sufficient for providing evidence. Temporal information and GOP are stored and used for this reasoning. The paper also evaluates its effectiveness across various benchmarks.

Critical strengths
  • The paper proposes a practical approach for long-video understnding
  • The paper’s key idea is using GOP structures in agentic pipeline, which is quite reasonable.
  • The paper reports performance improvements across multiple benchmarks.
Critical weaknesses
  • Fundamentally, the paper’s methodological contribution seems closer to an engineering optimization for efficient long-video understanding. That is, rather than introducing a new/interesting concept or proposing a novel paradigm, the framework seems to achieve performance improvements based on integration of existing components, which is not likely to interest ICLR audience.
  • Overall writing is not clear.
  • The performance gains are quite small.
  • The initial stage of the algorithm (i.e., sparse global inspection) is likely to miss the brief actions. Once these are skipped at this stage, the framework cannot recover them, which is critical.
Questions for authors
  • How does the proposed framework address brief actions that are missed/skipped from the sparse global inspection? Can this initial sampling stage rather exacerbate the long video understanding task?
  • Furthermore, as the video sequence becomes longer, the probability of missing relevant events increases. How does the method handle this issue?
  • The fundamental novelty of the paper needs to be demonstrated more clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.