- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AVA-Encoder, an agentic framework that converts videos into structured representations for downstream video agents. The representation combines film-level context, shared entity registries, shot-level descriptions, keyframe descriptions, and multimodal assets in a knowledge graph (KG).
The framework uses two optimization loops while keeping all foundation-model weights frozen. An outer loop updates natural-language encoding policies using reconstruction feedback from six training videos. An inner loop refines the KG for each input video, including at inference time, through iterative reconstruction, evaluation, and revision.
Critical strengths
- Relevant problem and useful representation design. Agent-oriented video representations are a worthwhile research direction. The hierarchical decomposition and shared entity registries provide a coherent way to organize information across shots.
- Explicit information bottleneck. Reconstruction is constrained to use textual descriptions derived from the input video, without passing source pixels to the generators. This makes reconstruction a meaningful, although incomplete, test of the information retained by the representation.
- Substantial improvements under the reported evaluation
Critical weaknesses
- The inner-loop shot reward appears to favor omission in some cases.The shot-refinement reward excludes absent facts from the scoring denominator. By contrast, the final evaluation assigns zero credit to missing facts. The treatment of missing critical facts also appears to differ between the two definitions. (Appendix D.2, p.21; Appendix B, p.17)
- The experiments do not fully isolate the novel contribution or control inference cost. Natural-language policy optimization, reconstruction feedback, and iterative critique are related to existing approaches such as TextGrad and Soap2Soap. The paper’s strongest potential contribution appears to be their integration with hierarchical video encoding and a reusable graph representation. The experiments should more directly isolate that contribution. (Related Work and §4)
- The experiments do not establish that the gains come from the graph representation rather than richer video descriptions. Hierarchical analysis and graph representation are separate design choices.
- Much of the advantage exists before dual-loop optimization, leaving its distinctive contribution insufficiently established in Table 2. The hierarchical encoder already exceeds the strongest external baseline by 14.1 points without either optimization loop.
Questions for authors
- How is audio reconstructed when the learned shot policy explicitly removes audio-related content from the video-generation prompt? Is there a separate audio-generation or conditioning pathway?
- Does the human-agreement study include comparisons between different methods, or only candidates from optimization trajectories? What are the inter-annotator agreement and agreement with final system rankings?
- What are the total generation calls, judge calls, token usage, runtime, and monetary cost per evaluated video? Do the gains remain against baselines given comparable inference budgets?
2 comments
Response to Reviewer VYcB (Answers of Questions)
[Q1: Audio-related content in reconstruction prompt ]
We sincerely apologize for including an incorrect round version of the system prompt in the appendix. During self-evolution, the agent identified the audio generation issue and updated the system prompt to require audio generation. We have updated the PDF with the correct final system prompt from the sixth evolution round (see Appendix N).
[Q2: Human Evaluation]
We conducted two additional human evaluation studies on our full benchmark (18 videos, 129 shots, and 246 keyframes), covering 4 methods (including 3 baselines). We randomly sampled one (method, GT) pair per shot/keyframe, yielding 129 shot pairs and 246 keyframe pairs.
-
VLM-generated questions: Two human annotators independently scored all pairs using the same questions and scoring rules as the VLM judge. Their averaged scores achieved 97.91% agreement with the VLM scores.
-
Human-generated questions: Two human annotators designed questions for each GT based on the same eight dimensions and question-design rules as the VLM judge, then scored the pairs. Their averaged scores achieved 89.32% agreement with the VLM scores.
[Q3: Generation Calls, Judge Calls, Token Usage and Runtime]
We randomly sampled 5 videos and measured the average cost per video:
- Without inner-loop refinement: 10.2 video understanding calls, 7.2 image understanding calls, 17.4 image generation calls, and 7.4 video generation calls, with 751,666.4 input tokens, 560,822.2 output tokens, and 32.7 minutes of runtime.
- With inner-loop refinement: An additional 19.2 video understanding calls, 50.2 image understanding calls, 49.4 image generation calls, and 6 LLM calls, with an average runtime of 89.2 minutes.
We acknowledge that our method uses more API calls than the baselines. However, as shown in Response 3, higher token usage does not necessarily lead to better reconstruction quality. Our gains mainly come from the method design, rather than simply increasing tokens or API calls.
Response to Reviewer VYcB
We thank the reviewer for the sightful questions.
1. [Excluded absent facts from the scoring denominator]
We sincerely apologize for this error in the manuscript. We confirmed that absent facts are included in the scoring denominator for both the inner-loop shot reward and the final evaluation across all reported results in the paper. We have corrected the corresponding description in Appendix D.2 to accurately reflect the implementation. We thank the reviewer for identifying this inconsistency.
2. [Novel Contributions and Cost Control]
Our novel contributions:
Differences from prior work:
Ablations of Method Contribution (Table 2): Single-level encoder validates the hierarchical encoder and KG; w/o both loops, w/o policy pseudo-training, and w/o KG refinement isolate each loop; w/o inner-loop gate, w/o outer-loop gate, and w/o both gates demonstrate the gates' contributions to generalization. Since the KG is directly produced by the hierarchical encoder, their contributions cannot be fully separated, but we ablate our system design components wherever possible.
Cost control: Section 5.1 and Appendix G limit each shot to 5 new reference keyframes and 1,200 video prompt tokens. Detailed API call and token usage statistics are provided in the following response.
3.[Gains from KG representation or richer video]
We added an experiment comparing rich video descriptions covering the same eight representation dimensions with our original pipeline. We evaluated reconstructions of 30 shots from 5 videos, comparing the token count per shot and reconstruction quality using the same metrics as Table 1 and Section 5.1. Results show that even with over 700% more tokens, rich video descriptions still underperform our representation. This confirms that reconstruction consistency requires carefully designed representations, rather than simply richer descriptions.
4. [Contribution of Dual-Loop Self-Evolution]
First, our dual-loop self-evolution enables several important capabilities not previously available:
Second, the strong performance before dual-loop optimization also demonstrates the effectiveness of our hierarchical encoder and KG representation design.