acceptodds
Withdrawn from ICLR 2027

AVA-Encoder: Towards Agent-Native Video Representation Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic video creation systems remain far from cinematic-grade quality: their underlying models struggle to plan how story, characters, visuals, and sound fit together, and high-quality creation trajectories to learn such planning abilities are scarce. Finished human films embody this knowledge, but in a tightly coupled audiovisual form that agents cannot directly read or manipulate. We propose the Agentic Video Auto-Encoder (AVA-Encoder), which works backward from finished films, turning them into agent-native representations with reconstruction itself as supervision. It encodes a film into a Film Knowledge Graph (Film KG) of structured-text nodes, typed dependency edges, and linked multimodal assets, then regenerates the film from this graph with a fixed decoder. Reconstruction residuals are distilled into textual gradients that drive dual-loop self-evolution: an outer loop learns a shared encoding policy across videos, and an inner loop refines each video's Film KG at test time. AVA-Encoder improves reconstruction fidelity by 20.7 percentage points over the strongest baseline; its learned policy outperforms a finely human-tuned policy with over 70% fewer system-prompt tokens; and Film KGs support coordinated editing and improve all tested downstream creation systems. We release the framework, a reconstruction benchmark, and a Film KG dataset.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Video Auto-Encoder, Video Representation Learning, Self-evolve Agents, Agentic Learning, Test-time Training, Knowledge Graphs, Video Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VYcBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AVA-Encoder, an agentic framework that converts videos into structured representations for downstream video agents. The representation combines film-level context, shared entity registries, shot-level descriptions, keyframe descriptions, and multimodal assets in a knowledge graph (KG).

The framework uses two optimization loops while keeping all foundation-model weights frozen. An outer loop updates natural-language encoding policies using reconstruction feedback from six training videos. An inner loop refines the KG for each input video, including at inference time, through iterative reconstruction, evaluation, and revision.

Critical strengths
  1. Relevant problem and useful representation design. Agent-oriented video representations are a worthwhile research direction. The hierarchical decomposition and shared entity registries provide a coherent way to organize information across shots.
  2. Explicit information bottleneck. Reconstruction is constrained to use textual descriptions derived from the input video, without passing source pixels to the generators. This makes reconstruction a meaningful, although incomplete, test of the information retained by the representation.
  3. Substantial improvements under the reported evaluation
Critical weaknesses
  1. The inner-loop shot reward appears to favor omission in some cases.The shot-refinement reward excludes absent facts from the scoring denominator. By contrast, the final evaluation assigns zero credit to missing facts. The treatment of missing critical facts also appears to differ between the two definitions. (Appendix D.2, p.21; Appendix B, p.17)
  2. The experiments do not fully isolate the novel contribution or control inference cost. Natural-language policy optimization, reconstruction feedback, and iterative critique are related to existing approaches such as TextGrad and Soap2Soap. The paper’s strongest potential contribution appears to be their integration with hierarchical video encoding and a reusable graph representation. The experiments should more directly isolate that contribution. (Related Work and §4)
  3. The experiments do not establish that the gains come from the graph representation rather than richer video descriptions. Hierarchical analysis and graph representation are separate design choices.
  4. Much of the advantage exists before dual-loop optimization, leaving its distinctive contribution insufficiently established in Table 2. The hierarchical encoder already exceeds the strongest external baseline by 14.1 points without either optimization loop.
Questions for authors
  1. How is audio reconstructed when the learned shot policy explicitly removes audio-related content from the video-generation prompt? Is there a separate audio-generation or conditioning pathway?
  2. Does the human-agreement study include comparisons between different methods, or only candidates from optimization trajectories? What are the inter-annotator agreement and agreement with final system rankings?
  3. What are the total generation calls, judge calls, token usage, runtime, and monetary cost per evaluated video? Do the gains remain against baselines given comparable inference budgets?
2 comments
AuthorsOpenReview ↗

Response to Reviewer VYcB

We thank the reviewer for the sightful questions.

1. [Excluded absent facts from the scoring denominator]

We sincerely apologize for this error in the manuscript. We confirmed that absent facts are included in the scoring denominator for both the inner-loop shot reward and the final evaluation across all reported results in the paper. We have corrected the corresponding description in Appendix D.2 to accurately reflect the implementation. We thank the reviewer for identifying this inconsistency.


2. [Novel Contributions and Cost Control]

  • Our novel contributions:

    1. An agent-native KG-based video representation for downstream agentic video editing and creation.
    2. A self-evolving representation learning paradigm using video reconstruction consistency as feedback. Our dual-loop evolution generalizes better to unseen videos than human-tuned policies, with fewer system prompt tokens.
  • Differences from prior work:

    1. Soap2Soap is a video reconstruction pipeline that reconstructs each video once without learning a representation policy. We instead learn a generalizable representation policy from reconstruction errors on a few videos.
    2. TextGrad provides general natural-language feedback optimization. We extend it with video reconstruction-consistency feedback to drive dual-loop self-evolution.
    3. Unlike standard iterative critique, that mainly improves for current input data, our method optimizes current-video reconstruction while preventing forgetting hisorical knowledge and generalizing improvements to unseen videos.
  • Ablations of Method Contribution (Table 2): Single-level encoder validates the hierarchical encoder and KG; w/o both loops, w/o policy pseudo-training, and w/o KG refinement isolate each loop; w/o inner-loop gate, w/o outer-loop gate, and w/o both gates demonstrate the gates' contributions to generalization. Since the KG is directly produced by the hierarchical encoder, their contributions cannot be fully separated, but we ablate our system design components wherever possible.

  • Cost control: Section 5.1 and Appendix G limit each shot to 5 new reference keyframes and 1,200 video prompt tokens. Detailed API call and token usage statistics are provided in the following response.


3.[Gains from KG representation or richer video]

We added an experiment comparing rich video descriptions covering the same eight representation dimensions with our original pipeline. We evaluated reconstructions of 30 shots from 5 videos, comparing the token count per shot and reconstruction quality using the same metrics as Table 1 and Section 5.1. Results show that even with over 700% more tokens, rich video descriptions still underperform our representation. This confirms that reconstruction consistency requires carefully designed representations, rather than simply richer descriptions.

MethodV ↑KF ↑V-BC ↑KF-BC ↑Overall ↑Prompt Tokens / Shot ↓
Rich Descriptions(RD)0.4280.5250.2730.2910.3793,426.5
Ours0.5520.7010.3770.3990.507410.7
Δ (Ours − RD)+0.124+0.176+0.104+0.108+0.128−88.0%

4. [Contribution of Dual-Loop Self-Evolution]

  • First, our dual-loop self-evolution enables several important capabilities not previously available:

    • Higher performance ceiling: The encoding policy learned by the outer loop outperforms the best human-tuned encoding system prompt, both with and without inner-loop refinement, while using over 73% fewer tokens. This gives higher performance ceiling to the whole system.
    • Autonomous Learning from User Data: The outer loop allows users to evolve policies tailored to their own video data and scenarios, without any human intervention.
    • Video-specific refinement: The inner loop enables test-time refinement to better capture the specific details of each input video.
  • Second, the strong performance before dual-loop optimization also demonstrates the effectiveness of our hierarchical encoder and KG representation design.

AuthorsOpenReview ↗

Response to Reviewer VYcB (Answers of Questions)

[Q1: Audio-related content in reconstruction prompt ]

We sincerely apologize for including an incorrect round version of the system prompt in the appendix. During self-evolution, the agent identified the audio generation issue and updated the system prompt to require audio generation. We have updated the PDF with the correct final system prompt from the sixth evolution round (see Appendix N).


[Q2: Human Evaluation]

We conducted two additional human evaluation studies on our full benchmark (18 videos, 129 shots, and 246 keyframes), covering 4 methods (including 3 baselines). We randomly sampled one (method, GT) pair per shot/keyframe, yielding 129 shot pairs and 246 keyframe pairs.

  1. VLM-generated questions: Two human annotators independently scored all pairs using the same questions and scoring rules as the VLM judge. Their averaged scores achieved 97.91% agreement with the VLM scores.

  2. Human-generated questions: Two human annotators designed questions for each GT based on the same eight dimensions and question-design rules as the VLM judge, then scored the pairs. Their averaged scores achieved 89.32% agreement with the VLM scores.


[Q3: Generation Calls, Judge Calls, Token Usage and Runtime]

We randomly sampled 5 videos and measured the average cost per video:

  • Without inner-loop refinement: 10.2 video understanding calls, 7.2 image understanding calls, 17.4 image generation calls, and 7.4 video generation calls, with 751,666.4 input tokens, 560,822.2 output tokens, and 32.7 minutes of runtime.
  • With inner-loop refinement: An additional 19.2 video understanding calls, 50.2 image understanding calls, 49.4 image generation calls, and 6 LLM calls, with an average runtime of 89.2 minutes.

We acknowledge that our method uses more API calls than the baselines. However, as shown in Response 3, higher token usage does not necessarily lead to better reconstruction quality. Our gains mainly come from the method design, rather than simply increasing tokens or API calls.

Reviewer_6ccVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that agentic video systems cannot learn cinematic planning because high-quality creation trajectories are scarce, while finished films contain that knowledge in a form agents cannot read or edit. AVA-Encoder is an agentic auto-encoder: a hierarchical VLM encoder (film, shot, keyframe) turns a video into a Film Knowledge Graph of structured-text nodes, typed edges, and generated assets; a frozen decoder (Nano Banana Pro + HappyHorse 1.0) must rebuild the video from that graph, with no source pixels passed to the generators. Reconstruction errors are turned into textual gradients. An outer loop rewrites the shared shot/keyframe prompts on 6 videos; an inner loop edits the test video’s graph, with acceptance gates meant to limit regressions.

On 18 held-out clips (129 shots, 246 keyframes), a Gemini fact-checklist protocol reports 49.0% Overall versus 28.3% for soap2soap. Ablations attribute most of the internal gain to hierarchical encoding (27.5% to 42.4% with both loops off); the two loops together add 6.6 points. A learned prompt beats a longer human-written prompt by 1.4 points with about 70% fewer tokens. The graph is also shown in a qualitative character-replacement edit, and pasting the full Film KG text into four existing creation agents raises a reference-free Gemini quality mean.

Critical strengths
  1. The task constraint is the right one for the claim. Forcing every system through a text graph and a shared decoder, and banning source pixels as generation references, measures whether an agent-readable representation actually retains reconstructable content. That is more informative than captioning or retrieval QA. The hierarchy ablation is large and interpretable: with both loops disabled, Overall moves from 27.5% (single-level) to 42.4%, and 42.4% still exceeds soap2soap at 28.3%. The gate ablations also move in the expected direction (49.0% to 44.9%, 46.5%, and 43.5%).

  2. The paper separates, at least on paper, the optimization signal from the reported metric, and it states the decoder interface budget (at most 5 reference keyframes and 1,200 shot-prompt tokens). The internal policy comparison is specific: the pseudo-trained shot prompt is 8,052 tokens versus 31,336 for the human-tuned prompt, and the keyframe prompt is 4,062 versus 13,574.

Critical weaknesses
  1. The headline 20.7-point gain is not identified by the current evaluation. Reval and the inner-loop shot reward are the same kind of Gemini checklist: eight dimensions, match / partial / mismatch, and a 0.4 cap when a critical fact fails. The same model family therefore both searches and scores. The reported 97.3% human agreement does not validate Table 1. Appendix B.6 uses 730 blinded triples from self-optimization runs on 10 videos; two annotators say which candidate is closer to the source, and no inter-annotator agreement or adjudication rule is given. That is not a blinded ranking of the four systems on the 18-video benchmark, and it does not check the absolute dimension scores. The test clips are canonical copyrighted scenes (Spirited Away, Joker, Titanic, Forrest Gump, Friends, Kung Fu Panda, and others), so Qwen, Gemini, and the generators may be recalling the scene rather than reading the graph. There is no per-video variance, no seed, and no ceiling from a human-written prompt under the same decoder. Full AVA also spends extra test-time search (up to 4 rounds per keyframe and 3 per shot). The shared budget limits only the decoder prompt, not the number of generations. The back-caption scores (V-BC 29.7%, KF-BC 34.6%) are far below direct V/KF (57.8% / 73.7%), so the 49.0% average is pulled up by the more lenient direct comparison. Style and motion remain weak even for AVA (V-style 39.5%, V-motion 43.7%).

  2. The use that motivates the paper is not tested. Film KGs are offered as reverse-engineered creation trajectories that could train planners, but no planner is trained. Downstream support is a reference-free Gemini mean on an unstated number of cases. Overall rises for all four systems, but Anim-Director’s Plot falls from 2.68 to 1.55 and VideoStudio’s Character falls from 3.00 to 1.43; VideoStudio’s Overall gain is only +0.15. Editing is a qualitative showcase (Figure 4) with no success rate and no measure of unintended changes. External baselines are transcript, storyboard, and remake systems retargeted to this text bottleneck, so much of the 20.7-point gap is “a pipeline built for this metric versus systems that were not,” while the actual self-evolution gain on the authors’ own encoder is 6.6 points with no uncertainty.

Questions for authors
  1. What automatic score is compared with the 710/730 human judgments: Reval, the shot-reward checklist, or the pairwise Closer judge used by the keyframe gate? What is the agreement between the two annotators, and how are disagreements resolved? Please also report human pairwise rankings among AVA-Encoder, soap2soap, VideoAnalyzer, and Storyboard Studio on the 18 evaluation videos, using the same shots that enter Table 1.

  2. How many image and video generations does full AVA-Encoder use per evaluation video, relative to each baseline? Please add a comparison in which every method gets the same number of decoder calls. Separately, please report per-video means and a uncertainty estimate, and repeat the evaluation on clips that are unlikely to appear in Qwen, Gemini, Nano Banana, or HappyHorse training data. A human-written prompt under the same decoder would give the missing ceiling.

  3. For Table 3, how many source stories and generated videos are scored, and what user request is given besides the Film KG? Is the injected graph the reverse encoding of a specific target film? Given the Plot drop for Anim-Director and the Character drop for VideoStudio, do human raters reproduce the Gemini means?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6ccV

We thank the reviewer for the questions.

[1.Inner-loop shot reward and Evaluation Method]

First, the inner-loop reward and R_eval use different QA lists, rather than identical scoring criteria. Across 5 randomly sampled videos, their average QA overlap is only 30.75% for shots and 14.05% for keyframes. They share the same eight dimensions because both measure reconstruction consistency, while different QA criteria help prevent direct reward hacking.

Second, we replaced Gemini with Qwen 3.7 Plus and Qwen 3.8 Max as the R_eval judge, achieving 98.27% and 99.13% agreement with Gemini, respectively. This confirms the stability of our evaluation across model families. Our evaluation measures factual QA consistency against ground-truth videos/keyframes, rather than subjective visual preferences, making it largely independent of the judge model subjective preferences.

[2.Human Evaluation]

We conducted two additional human evaluation studies on our full benchmark (18 videos, 129 shots, and 246 keyframes), covering 4 methods (including 3 baselines). We randomly sampled one (method, GT) pair per shot/keyframe, yielding 129 shot pairs and 246 keyframe pairs.

  1. VLM-generated questions: Two human annotators independently scored all pairs using the same questions and scoring rules as the VLM judge. Their averaged scores achieved 97.91% agreement with the VLM scores.

  2. Human-generated questions: Two human annotators designed questions for each GT based on the same eight dimensions and question-design rules as the VLM judge, then scored the pairs. Their averaged scores achieved 89.32% agreement with the VLM scores.

3. [Memorization Concerns and Evaluation on New Films]

First, we replace all identifiable IP references (e.g., naming character as “a middle-aged man”) before sending prompts to diffusion models, preventing generators from identifying the original films through these references. Moreover, diffusion models themselves reject image and video generation requests containing copyrighted IP references to protect intellectual property.

Second, we added experiments on 5 films released in November 2026 (Sea Star, PIKA!, Liminal, Doger, and Gatorville), using the same evaluation metrics as Table 1 and Section 5.1. The average scores are 0.575 (V), 0.736 (KF), 0.324 (V-BC), 0.362 (KF-BC), and 0.499 (Overall).

Even if Qwen and Gemini have learned the plots of classic films, plot knowledge alone is insufficient to reconstruct precise visual details from sparse information. Reconstruction still relies heavily on the detailed visual information and dependencies stored in our KG representation.

4. [Planner Training with Agentic Trajectories]

We added experiments on using the back-engineered agentic trajectories of the dataset to train agents with better video creation planning. We sampled 145 videos and 7,863 agentic video creation trajectories to fine-tune Qwen3-8B for 3 epochs and evaluate it on 5 independent test cases using the same protocol as Sec. 5.4. It improves Overall video creation quality by 38.5%.

DimensionBaseTrajectory SFTΔ
Character3.173.00−0.17
Plot / Story1.753.00+1.25
Camera3.103.20+0.10
Style2.254.00+1.75
Audiovisual1.252.75+1.50
Overall2.303.19+0.89 (+38.5%)

5. [Downstream Video Creation Few-shot Settings]

Evaluation setup: We evaluated 10 video creation cases. Each user request was: "Create a complete Oscar-quality short film based on [theme]." For each case, we selected the top-1 most relevant KG from our 500-KG dataset by matching theme embeddings. The same retrieved KG was provided as a few-shot reference to all agentic video creation systems for generation and evaluation, rather than using the reverse encoding of a specific target film.

Human agreement: We added human evaluations by two annotators, achieving 90.20% overall agreement and 83.31% plot-level agreement with Gemini scores.

Dimension-level variations: We attribute variations across dimensions mainly to differences between agent frameworks and the instability of single-KG few-shot conditioning. Nevertheless, the overall improvements across frameworks demonstrate the quality, agent-readability, and downstream usefulness of our KG representation.

AuthorsOpenReview ↗

Response to Reviewer 6ccV

6. [Gains Beyond Test-Time Search and Across Metrics]

  1. Test-time search is not the main source of improvement: Without inner-loop KG refinement, AVA still achieves 45.8% Overall, outperforming the strongest baseline (28.3%) by 17.5 pp.

  2. Lower back-caption scores do not imply smaller gains: AVA improves V-BC and KF-BC over Soap2Soap by 13.9 pp and 13.3 pp, respectively, averaging +13.6 pp even when considering only these two metrics.

  3. Style and motion also improve substantially: V-Style increases from 28.2% to 39.5% (+11.3 pp), and V-Motion from 30.2% to 43.7% (+13.5 pp).

7. [Controlled Editing and Unintended Changes.]

Editing is not merely a qualitative showcase. Sec. 4.2 (Eq. 6) and Appendix L.2 (Table 9) explicitly define edit-conditioned dependency propagation, restricting updates to the affected subgraph while preserving all unrelated KG nodes. Figure 4 and Appendix L further demonstrate coordinated multi-shot editing with preservation of unrelated content. We acknowledge that these qualitative results do not establish a quantitative success rate or guarantee pixel-level invariance in regenerated videos.

8. [Decoder-Call Budget Comparison]

On our 18-video benchmark, the average generation calls per video are:

  • AVA (without inner loop): 7 video + 16 image generations.
  • Soap2Soap: 7.5 video + 17.5 image generations.
  • VideoAnalyzer: 7 video + 11 image generations.
  • Storyboard Studio: 7 video + 10 image generations.

Thus, all methods already use comparable video-generation budgets, and AVA uses fewer total generation calls than Soap2Soap. Importantly, without inner-loop refinement, AVA still achieves 45.8% Overall, outperforming the strongest baseline (28.3%) by 17.5 pp. This demonstrates that our main gains do not rely on additional test-time generation.

9. [Baseline Fairness and Isolated Contributions]

  • Fair Baseline Comparison. As detailed in Appendix G, we preserve each baseline's original analysis and representation, adapting only its output to the shared reconstruction decoder. All methods use identical understanding/generation backbones, decoder-interface budgets, and source-pixel exclusion constraints. Thus, the comparison evaluates how effectively different representations preserve information for reconstruction, rather than differences in generation backbones.

  • Contributions Beyond Self-Evolution. The 20.7-point gain is an end-to-end improvement, not attributed solely to self-evolution. Table 2 explicitly isolates both contributions: hierarchical encoding improves 27.5% → 42.4% (+14.9 points) without either optimization loop, while dual-loop self-evolution further improves 42.4% → 49.0% (+6.6 points). Importantly, even without self-evolution, AVA-Encoder achieves 42.4%, outperforming the strongest external baseline (28.3%) by 14.1 points. These ablations demonstrate that the gain is not merely attributable to iterative optimization.

10. [Human-Written Prompt Baseline.]

We agree that per-instance human-written prompts could provide a reference for decoder performance, but our goal is automatic representation learning without human-authored prompts. We already compare against a human-tuned encoding policy (Appendix F), which achieves 44.4% Overall, versus 45.8% for our learned policy without test-time refinement (Table 2), using the same frozen decoder. While not a per-instance decoder ceiling, this comparison demonstrates the effectiveness of automated policy learning over human prompt engineering.

Reviewer_1XMEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes AVA-Encoder, a framework that reverse-engineers a finished film into a Film Knowledge Graph (Film KG). and then regenerates the film from this graph with a frozen image/video generator. The encoder is a three-level (film → shot → keyframe) LLM pipeline; faithfulness is optimized by converting reconstruction failures into natural-language critiques that revise either a shared encoding policy (outer loop) or the per-video KG (inner loop), with acceptance gates. Reported Overall reconstruction fidelity is 49.0%, +20.7pp over the strongest of three compared systems.

Critical strengths
  • The paper propose a concrete 3-level Film KG schema with ten node types, 11 relation types an explicit asset layer, and a defined edit operator.
  • Compare to baseline, the paper's method achieves +20.7pp and is the strongest among all baselines.
Critical weaknesses
  • The writing of this paper is quite confusing regard of its concept. The title and abstract advertise "representation learning," "Auto-Encoder," a policy that "learns," and "dual-loop self-evolution." But in fact no parameters are updated anywhere and all model calling are through api.
  • The released "benchmark" and dataset are underspecified, and the dataset is never used. The Film KG dataset listed in its third contribution contains very few detail and the actual dataset used for contribution number 2 is merely 18 videos with 6 videos for pseudo training.
  • Qualitative evidence does not supports the fidelity claim. In the appendix comparisons (Fig. 5–10) the baselines frequently fail for reasons unrelated to representation quality both Storyboard Studio and VideoAnalyzer produce photorealistic output, a style/prompt mismatch rather than an information loss.
Questions for authors

Since no weights are updated, what distinguishes "pseudo-training" from ordinary prompt search, and what is the intended meaning of "Auto-Encoder" beyond a task-level analogy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1XME

We sincerely thank the reviewer for the careful reading and valuable questions.

1. [Learning without weight updates.]

In Agentic Test-Time Self-Evolution, agents learn by updating prompts, memory or skills, without parameter updates. Examples include:

  • Prompt/Policy Evolution: EvoTest (He et al., ICLR 2026), Promptbreeder (Fernando et al., ICML 2024), and GEPA (Agrawal et al., ICLR 2026).
  • Memory/Skill Evolution: Skill-Pro (Mi et al., ICML 2026), MemEvolve (Zhang et al., ICML 2026), Mem²Evolve (Cheng et al., ACL 2026), and Dynamic Cheatsheet (Suzgun et al., EACL 2026).
  • Verbal Feedback and Self-Refinement: Reflexion (Shinn et al., NeurIPS 2023) and Self-Refine (Madaan et al., NeurIPS 2023).

2. [Clarification of Terms]

  • By "Learning", we mean "In-Context Learning", rather than model parameter update. Language Agents can evolve the 3-layer system prompts at test time within the hierarchical agentic video encoder.
  • Our "Representation Learning" uses explicit Knowledge Graph as representation, rather than implicit embeddings that agents cannot directly understand or edit.
  • By "Agentic Encoding", we mean mapping videos into KG representations through a hierarchical video understanding module.

[Q1:How Pseudo-Training differs from ordinary prompt search.]

Prompt search updates prompts for one specific seen input, it describes how we propose candidate updates, but our goal is to learn an encoding system prompt that generalizes across unseen videos, not optimizing a prompt for one seen video. We use Historical Replay and an Anti-Forgetting Gate to maintain performance across previously learned videos. With only 6 pseudo-training videos, the self-evolved agentic encoding prompt improves reconstruction fidelity on 18 unseen videos from 42.4% to 45.8%, outperforming the human-tuned policy (44.4%) while using over 70% fewer system-prompt tokens.

[Q2: The intended meaning of "Auto-Encoder" beyond a task-level analogy.]

"Agentic Auto-Encoder" is our agentic test-time self-evolution method; It is built upon the core concept of traditional Auto-Encoder, with reconstruction error serving as the learning signal. It drives our agentic self-evolution of the KG representation constructing method that generalizes for unseen videos. To adapt to test-time self-evolving language agents, we use structured KG representation to replace embedding representation, and use Textual Gradient to replace parameter updates.

Our main idea is to extend reconstruction-based representation learning from neural weight updates to agent-operable policies and representations, allowing agents to learn generalizable encoding policies and refine video representations at test time.


3. [Clarification of the Film KG Dataset and Benchmark]

The Film KG Dataset (500 videos) and the Reconstruction Benchmark (18 eval videos) are 2 separate datasets.

  • The Reconstruction Benchmark evaluates our reconstruction performance through 18 unseen tested videos in Table 1. It uses fine-grained multidimensional vlm-as-a-judge and aligned human evaluation, following Ref4D-VideoBench (Wei et al., CVPR 2026), DramaAgent (Huang et al., AACL-IJCNLP 2026), MovieBench (Wu et al., CVPR 2025), and ViStoryBench (Zhuang et al., CVPR 2026).

  • The KG dataset consists of synthetic data produced by our method. Built upon 500 2~20min high-quality films (Appendix H.1, L1307–1320) covering over 20K shots and 50K keyframes, the dataset includes each video's (1)Knowledge Graph Representation and (2)back-engineered agentic training trajectories. Both of them are used to help improve downstream task performance:

    • Agentic Video Topological Editing: KG allows agents to directly edit nodes and propagate edits through graph topology, enabling coordinated edits across any video dimension, such as the character and style coordinated edits performance shown in Figures 4, 14, and 15.
    • Agentic Video Creation via Few-Shot: Table 3 shows our KG improves all 4 open-sourced agentic video creation systems through simple KG few-shot input, with an +23.97% averaged Overall improvement.
    • Agentic Video Creation via Trajectory Training: We added experiments on using the back-engineered agentic trajectories of the dataset to train agents with better video creation planning. We sampled 145 videos and 7,863 agentic video creation trajectories to fine-tune Qwen3-8B for 3 epochs and evaluate it on 5 independent test cases using the same protocol as Sec. 5.4. It improves Overall video creation quality by 38.5%. | Dimension | Base | Trajectory SFT | Δ | |---|---:|---:|---:| | Character | 3.17 | 3.00 | −0.17 | | Plot / Story | 1.75 | 3.00 | +1.25 | | Camera | 3.10 | 3.20 | +0.10 | | Style | 2.25 | 4.00 | +1.75 | | Audiovisual | 1.25 | 2.75 | +1.50 | | Overall | 2.30 | 3.19 | +0.89 (+38.5%) |
AuthorsOpenReview ↗

References

[1] He et al. (2026). EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems. International Conference on Learning Representations (ICLR 2026).

[2] Shinn et al. (2023). Reflexion: Language Agents with Verbal Reinforcement Learning. Advances in Neural Information Processing Systems 36 (NeurIPS 2023).

[3] Fernando et al. (2024). Promptbreeder: Self-Referential Self-Improvement via Prompt Evolution. Proceedings of the 41st International Conference on Machine Learning (ICML 2024).

[4] Agrawal et al. (2026). GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. International Conference on Learning Representations (ICLR 2026).

[5] Mi et al. (2026). Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026).

[6] Zhang et al. (2026). MemEvolve: Meta-Evolution of Agent Memory Systems. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026).

[7] Cheng et al. (2026). Mem²Evolve: Towards Self-Evolving Agents via Co-Evolutionary Capability Expansion and Experience Distillation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026), Volume 1: Long Papers.

[8] Suzgun et al. (2026). Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026), Volume 1: Long Papers.

[9] Madaan et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. Advances in Neural Information Processing Systems 36 (NeurIPS 2023).

[10] Wei et al. (2026). Ref4D-VideoBench: Four-Dimensional Reference-Based Evaluation of Text-to-Video Generative Models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2026).

[11] Huang et al. (2026). DramaAgent: Agentic Storytelling Video Generation. Proceedings of the 15th International Joint Conference on Natural Language Processing and the 5th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (AACL-IJCNLP 2026), Volume 1: Long Papers.

[12] Wu et al. (2025). MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2025).

[13] Zhuang et al. (2026). ViStoryBench: Comprehensive Benchmark Suite for Story Visualization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2026).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the Area Chair and reviewers for their feedback. Below are the main concerns and our responses.

Main concernReviewer 1XMEReviewer 6ccVReviewer VYcBOur response and evidence
1. Method and noveltyLearning/auto-encoding without weight updates?—Differences from TextGrad and Soap2Soap?We study self-evolving agents that learn encoding policies and KGs, not model weights. Using video reconstruction consistency as feedback, we extend the auto-encoder principle to agent-readable, editable representations, supporting both input-specific refinement and policies that generalize to unseen videos.
2. Gains and dual-loop value—How much comes from self-evolution?Representation design or richer descriptions?Encoder + KG: Without either loop, Overall is 42.4%, versus 27.5% for the single-level encoder and 28.3% for the strongest baseline.
Dual loop (i) — higher fidelity: The learned policy beats human tuning (45.8% vs. 44.4%) with over 70% fewer system-prompt tokens.
Dual loop (ii) — user-data learning: The outer loop autonomously learns generalizable policies from users’ videos.
Dual loop (iii) — input-specific refinement: The inner loop autonomously refines each film’s KG. Together, both loops reach 49.0% (+6.6 pp).
Rich-description control: On 30 shots from 5 videos, our method scores 50.7% vs. 37.9% with 88% fewer tokens.
3. Film KG dataset utilityDataset vs. benchmark; actual use?No planner training; limited downstream evidence.—Separate resources: The 500-video dataset supports creation, training, and editing; the 18-video benchmark evaluates reconstruction.
Few-shot: Theme-retrieved KGs improve Overall in all four systems on 10 cases, with a 23.97% mean relative gain. These are references, not target-film encodings.
Trajectory training: Qwen3-8B trained on 7,863 trajectories from 145 videos improves Overall from 2.30 to 3.19 (+38.5%) on 5 independent test cases.
Editing: Character/style edits propagate across linked shots while unrelated KG nodes remain unchanged (Figs. 4, 14–15; qualitative evidence).
4. Evaluation reliability—Judge bias; human validation; metric leniency.Human alignment across methods?Human–VLM agreement: 97.91% with VLM-written questions; 89.32% with human-written questions. Two annotators scored 375 pairs covering 18 videos and 4 methods, sampling one method per source shot/keyframe.
Judge checks: Two Qwen judges each show over 98% agreement with Gemini. Optimization and final evaluation use different QA lists.
Other checks: Back-caption metrics alone retain a 13.6 pp lead over soap2soap. Table 3’s human–Gemini Overall agreement is 90.20%.
5. New-film generalization—Memorized classic scenes?—Existing safeguard: Identifiable IP references were already replaced with visual descriptions before generation.
New-film test: 5 films released in November 2026 yield 49.9% Overall, close to Table 1’s 49.0%, supporting that performance is not solely due to memorized classics.
6. Fairness and costStyle/prompt mismatches?Baseline adaptation; extra search?Comparable budgets; actual cost?Shared setup: We preserve baseline pipelines and share understanding/generation models, with up to 5 generated reference keyframes and 1,200 prompt tokens per shot. Generators receive no source pixels.
Without inner-loop search: AVA averages 7 video + 16 image generations per video, versus soap2soap’s 7.5 + 17.5, while leading by 17.5 pp in Overall.
Cost disclosure: We report calls, tokens, and runtime, and acknowledge the extra cost of full refinement.

Appendix corrections. D.2 now states that absent facts were included in the denominator in all reported experiments. N now contains the correct final shot prompt, which requires audio generation.

pp = percentage points. The 23.97% and 38.5% gains are relative improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.