acceptodds
Under review as a conference paper at ICLR 2027

Frameworkers: A Dynamic Multi-Agent Framework for AI Video Production

Abstract

Modern video generators produce high-quality individual clips, but complete video production requires coordinating many interdependent steps, from scripting and storyboarding to generation and editing, together with the assets they pro duce and consume. Existing automated systems rely on fixed pipelines that adapt poorly to diverse inputs,while general-purpose large language models (LLMs)re main unreliable for long-horizon orchestration and multimodal asset routing. We introduce FRAMEWORKERS, which replaces the fixed production pipeline with learned, descriptor-conditioned orchestration over modular capabilities, grounded in a persistent Workspace. The Director treats production as dynamic task man agement, editing a Dynamic Task Stack to decide what to do next, which capabil ity to invoke, and how to replan the remaining work after feedback or failure. The Assistant executes each task against shared state that carries scripts, assets, and memory across steps. Capabilities are registered as sub-agents described only by natural-language descriptors, so adding one requires no change to the framework. We train the Director for this routing task with supervised fine-tuning followed by Group Relative Policy Optimization (GRPO). FRAMEWORKERS reaches 88.4% chain-level routing accuracy against 71.2% for the strongest LLM planner we evaluate,recovers from 96.7% of injected runtime failures with a general-purpose LLM as replanner and no recovery-specific training, correctly invokes sub-agents unseen in training, and is rated above single-agent and prior multi-agent video systems in automatic and human evaluation, with the largest human-rated gains on controllability and cross-shot consistency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computing methodologies → Multi-agent planning.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ExCCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FRAMEWORKERS, a multi-agent framework for automated video production. A Director dynamically plans and revises tasks, while an Assistant coordinates specialized sub-agents and manages scripts, reference assets, and generated content through a shared Workspace. Unlike a fixed pipeline, the framework can adapt its workflow to different user inputs and recover from intermediate failures. The Director is trained with supervised fine-tuning followed by GRPO to improve sub-agent selection and ordering. Experiments evaluate routing accuracy, failure recovery, integration of unseen sub-agents, and end-to-end video quality, reporting improvements over general-purpose LLM planners and existing video-production systems.

Critical strengths
  1. The proposed FrameWorkers replaces the fixed pipeline with orchestration composed at runtime, with dynamic task management, workspace-grounded execution, and learned routing, which is reasonable and interesting as the ai video production do not usually strictly follow the same and fixed structure.

  2. The performances of the multi-agent framework are impressive.

  3. The whole paper is well writen and easy to understand.

Critical weaknesses
  1. The technical novelty needs to be clearer. Although the paper discusses prior work and compares with MovieAgent, it remains unclear what the Dynamic Task Stack and Workspace add beyond existing multi-agent planning and coordination mechanisms. For example, Magentic-One[1] already supports dynamic task delegation, progress tracking, and replanning after failures, while Plan-and-Act[2] separates planning from execution and trains the planner. A more direct comparison would help clarify which mechanisms are new and which are adapted for video production.

[1] Adam Fourney et al. Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks. arXiv preprint arXiv:2411.04468, 2024.

[2] Lutfi Eren Erdogan et al. Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks. arXiv preprint arXiv:2503.09572, 2025.

  1. The generated videos in supplementary are only from FrameWorkers. There seems to be a lack of qualitative comparison with previous methods (only frame comparison in the main manuscript), which may weaken the effectiveness of the proposed method.

  2. Insufficient details and ablation results. (1) Insufficient details on maintaining cross-shot consistency: The paper mentions persistent identity anchors in the Workspace and evaluator-based rework, particularly in Figure 2 and the discussion of Figure 4. However, the method section does not clearly explain how these mechanisms maintain character/prop/scene consistency across shots. (2) Limited end-to-end ablation of Director training: Tables 1 and 2 show that SFT and GRPO improve routing accuracy. However, it remains unclear how much these improvements contribute to the final video quality (better to provide qualitative results).

Questions for authors
  1. About details on maintaining consistency: How are the reference assets created, selected, and passed to the generation models? How is visual drift detected, and what triggers regeneration? A more explicit description, together with ablations of reference reuse and evaluator-based rework, would help explain where the consistency gains come from.

  2. About ablation study: Could the authors compare pretrained, SFT-only, and SFT+GRPO Directors within the same framework, while keeping the Assistant, sub-agents, generation backends, and rework budget fixed? This would help show whether better routing leads to better instruction following and cross-shot consistency, beyond the gains provided by shared reference assets and evaluator-based rework.

10 comments
AuthorsOpenReview ↗

W1. Technical Novelty over Magentic-One and Plan-and-Act Part 1

We thank the reviewer for the careful evaluation of FrameWorkers and the positive assessment of our system design and experimental results. In response to the concerns regarding technical novelty, End-to-end Video Evaluation, cross-shot consistency, and Director Training, we further clarify the relevant technical design and provide additional experiments and analyses.

W1. Technical Novelty over Magentic-One and Plan-and-Act

FrameWorkers addresses dynamic task orchestration and cross-stage multimodal coordination in complex video production. Unlike general-purpose Agent Task Planning, video production involves interdependent stages such as scripting, storyboarding, character and scene references, KeyFrames, video, audio, and final composition. Each stage requires not only correct task scheduling but also specific multimodal assets generated in preceding stages. Moreover, generation failures, quality feedback, or changes in user requirements may necessitate revising subsequent work without re-executing costly generation steps that have already been completed.

To address these challenges, FrameWorkers combines three mechanisms: (1) a Learned Director, which dynamically selects production capabilities and plans execution dependencies based on user requirements; (2) a Dynamic Task Stack, which explicitly maintains task execution states and modifies pending tasks through Suffix Editing while preserving completed work; and (3) a Multimodal Workspace, which manages media assets generated across stages and dynamically resolves and passes the appropriate assets according to Sub-agent input requirements.

Comparison with Magentic-One and Plan-and-Act. The main design differences among the three frameworks are summarized below:

FrameworkPlanningExecution ManagementIntermediate Results
Magentic-OneAn Orchestrator coordinates general-purpose AgentsTask/Progress Ledgers track execution progress and support ReplanningManages general-purpose tool execution processes and task results
Plan-and-ActSeparates a learned Planner from an ExecutorThe Planner and Executor collaborate on long-horizon tasksExecutes planned steps through tool calls
FrameWorkersCapability Selection and Dependency-aware Routing based on Sub-agent DescriptorsThe Dynamic Task Stack maintains execution states and modifies pending tasks through Suffix EditingThe Multimodal Workspace manages cross-stage media assets and performs Semantic Asset Resolution

The central technical contribution of FrameWorkers is the integration of dynamic task planning, incremental execution, and multimodal asset coordination, allowing video-production workflows to adapt continuously to task requirements and runtime states without predefined fixed Pipelines for different tasks. This design requires not only selecting which Sub-agents to execute, but also maintaining task states during production, preserving existing results, and correctly organizing media-asset dependencies across stages.

Specifically, unlike Magentic-One, which manages task progress and coordinates execution through Task/Progress Ledgers, the Dynamic Task Stack in FrameWorkers organizes task states into a directly editable execution structure. The Director can adjust the pending suffix through ADD and DELETE, while the Workspace continually preserves artifacts produced by completed tasks, allowing the recovered production workflow to reuse existing results directly. At the same time, through a Strict-out, Loose-in Interface, the Workspace dynamically resolves the required inputs based on Sub-agent input requirements and semantic asset descriptions, without relying on fixed Producer–Consumer wiring. The Task Stack addresses how to incrementally modify the production workflow, while the Workspace addresses how tasks in the modified workflow correctly retrieve and reuse existing media assets. Together, they support continuously running multimodal video production.

Compared with Plan-and-Act, the Learned Director in FrameWorkers does not merely learn to generate task plans; it must also infer the required production capabilities and their dependencies from heterogeneous Sub-agent Descriptors, so that the resulting plans can be used directly for subsequent task execution and asset coordination. This design enables the Director to organize task-chain structures unseen during training, rather than being restricted to predefined production workflows. Tables 1 and 2 provide experimental evidence: on DCH, which contains unseen task-chain structures, the trained Director achieves 81.5% Chain Accuracy, an improvement of 33.5pp over the 48.0% achieved with SFT-only training.

AuthorsOpenReview ↗

W1. Technical Novelty over Magentic-One and Plan-and-Act Part 2

Furthermore, descriptor-conditioned planning enables FrameWorkers to extend its production capabilities without retraining the Director. The Selection Accuracy reported in Table 4 measures whether the Director correctly selects Sub-agents unseen during training based on task requirements, reaching 93.0% and 99.2% for ComedyAgent and ScreenwriterAgent, respectively. To further evaluate whether these new capabilities can be integrated into the complete production workflow, we additionally assess whether the task chains predicted by the Director correctly invoke the new Sub-agents and successfully complete the entire execution process. With runtime Replanning, the Joint Success rates reach 90.5% and 97.5% for ComedyAgent and ScreenwriterAgent, respectively. Table 8 in Appendix E further illustrates the diverse input configurations and output capabilities supported by the framework. These results demonstrate the design advantages of FrameWorkers: by dynamically selecting, organizing, and coordinating specialized Sub-agents, it supports different types of video-production tasks without redesigning a production workflow for each requirement.

To further validate the practical contributions of these mechanisms, we separately evaluate the recovery correctness of the Dynamic Task Stack, post-recovery execution performance, and the Workspace's cross-stage asset-resolution capability.

(1) Dynamic Task Stack Ablation

The Dynamic Task Stack improves the reliability of runtime failure recovery by explicitly maintaining execution states and modifying pending tasks. To validate this mechanism, we start from manually verified correct production chains and construct controlled perturbation cases by injecting three types of runtime failures: T1 (quality-check failure), T2 (one missing required upstream Producer), and T3 (two missing upstream Producers). These failures require the Replanner to restore the necessary production capabilities and their dependencies based on the current execution state. We construct 1,556 and 466 perturbed cases on DCE and DCH, respectively.

To distinguish the effects of execution-history visibility from those of structured task updates, we compare three recovery strategies. All methods use the same Replanning LLM, user goal, Agent Descriptors, and failure report; only the execution-state information available to the Replanner and the plan-update mechanism differ.

MethodInformation Available to the ReplannerPlan Update Strategy
Stateless Full RegenerationUser goal and failure report, without execution historyRegenerate the entire plan and execute from scratch
Magentic-One-style Log-aware Full RegenerationUser goal and an execution log containing the initial plan, completed steps, and failure recordsRegenerate the entire plan and execute from scratch
FrameWorkersCompleted, failed, and pending steps in the Task StackModify pending tasks through Suffix Editing while preserving completed artifacts

Here, Magentic-One-style Log-aware Full Regeneration is a controlled comparison strategy inspired by its Task/Progress Ledger approach, not a complete reproduction of the official Magentic-One system.

All methods are evaluated using the same recovery-correctness criterion: the recovered production chain must contain all required capabilities and satisfy Producer–Consumer dependencies.

We evaluate Gemini 2.5 Flash and Gemini 2.5 Pro as Replanners on DCE and DCH to measure Recovery Correctness under the different strategies. The task-chain structures in DCE appeared in the Director's training set, whereas DCH contains task-chain structures unseen during training. The values in parentheses indicate the number of controlled perturbation cases constructed by injecting T1–T3 failures. The Gemini 2.5 Pro experiment on DCE uses 600 of these cases.

AuthorsOpenReview ↗

W1. Technical Novelty over Magentic-One and Plan-and-Act Part 3

The percentages in the table denote Recovery Correctness: whether the task chain after recovery contains all required production capabilities and satisfies the corresponding Producer–Consumer dependency constraints.

MethodGemini 2.5 Flash / DCE (1,556)Gemini 2.5 Flash / DCH (466)Gemini 2.5 Pro / DCE (600)Gemini 2.5 Pro / DCH (466)
Stateless Full Regeneration88.4%86.5%89.7%90.6%
Magentic-One-style Log-aware Full Regeneration98.3%98.5%93.0%93.1%
FrameWorkers99.0%99.4%99.2%99.8%

FrameWorkers improves over Stateless Full Regeneration by 9.2–12.9pp across all four experimental settings, with all corresponding 95% CIs excluding zero. This demonstrates the importance of execution-history visibility and state management for reliable recovery.

Compared further with Log-aware Full Regeneration, FrameWorkers achieves gains of +0.7pp (95% CI [−0.1, 1.5]) and +0.9pp ([−0.4, 2.2]) with Gemini 2.5 Flash; neither difference is statistically significant. With Gemini 2.5 Pro, however, FrameWorkers achieves improvements of +6.2pp ([4.1, 8.4]) and +6.7pp ([3.7, 10.1]), respectively.

These results show that execution-history visibility already improves recovery performance substantially. Beyond this, the Task Stack organizes execution states into directly editable task records and updates the remaining production workflow through Suffix Editing, further improving recovery correctness with Gemini 2.5 Pro.

(2) Execution-level Validation

The Task Stack not only supports correct recovery planning but also improves task completion after failures and reduces redundant execution by reusing completed work. The preceding experiments primarily verify whether the recovered plans contain the required capabilities and satisfy dependency constraints, but they do not directly reflect actual execution outcomes and execution overhead after recovery. We therefore conduct additional Execution-level Validation.

Specifically, we construct 75 controlled failure cases, spanning T1, T2, and T3, with 25 cases per type. All methods use the same initial production plans and failure conditions, and run with the same Director, Assistant, Workspace, and task execution framework. Only the post-failure recovery strategy is varied.

Fixed Pipeline stops upon failure; Stateless Regeneration regenerates and executes the entire plan in a new Workspace; and FrameWorkers preserves completed artifacts in the original Workspace, modifies the remaining tasks through Suffix Editing, and continues execution. The latter two strategies allow up to two rounds of Replanning.

We evaluate three metrics: Completion, whether all required production capabilities are completed in a valid dependency order; Agent Executions / Case, the total number of Agent executions per case; and Re-executions / Case, the number of repeated executions of previously completed tasks.

MethodCompletionAgent Executions / CaseRe-executions / Case
Fixed Pipeline0%1.630
Stateless Regeneration90.7%8.611.65
FrameWorkers100%6.570.05

FrameWorkers achieves 100% Task Completion, compared with 90.7% for Stateless Regeneration. At the same time, average Agent Executions decrease from 8.61 to 6.57, and Re-executions decrease from 1.65 to 0.05.

These results validate the execution-level role of the Task Stack: by preserving completed tasks and their artifacts, FrameWorkers can continue executing the remaining work after failure recovery without re-executing the entire production workflow, thereby improving task completion and reducing execution overhead.

Reviewer_3pQzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

FrameWorkers addresses complete video production from heterogeneous user requests and assets. A Director plans calls to descriptor-defined sub-agents and edits pending work through a Dynamic Task Stack, while an Assistant resolves concrete inputs from a persistent Workspace, invokes sub-agents, checks outputs and reports execution state. Completed history is preserved while the unexecuted suffix can be revised. The initial routing policy is trained with supervised fine-tuning followed by GRPO, using format validity, capability coverage and dependency order as rewards. The implementation combines planning, media generation, asset management, evaluation and composition rather than introducing a new video-generation backbone.

Critical strengths
  1. The architecture addresses a real production problem and is described concretely. Optional steps, persistent media assets, changing requirements and failure recovery are legitimate requirements beyond isolated clip generation. The Director/Assistant separation, immutable completed history and descriptor interface provide a coherent implementation.
  2. The routing results are substantial within the stated evaluation.
  3. Real-video comparisons share generation backends within each benchmark, and the appendices provide useful training and evaluation details. The authors explicitly disclose mock execution, the recovery backend and weaker significance under per-rater analysis。
Critical weaknesses
  1. The central mechanism is not isolated. Initial routing, Gemini-based recovery and reference-chain execution with mock media are evaluated separately; no controlled video comparison holds assets, generation calls and evaluator budgets fixed while varying dynamic task editing or Workspace resolution in Section 3. Existing logs could clarify actual dynamic behavior, but establishing its independent contribution may require additional experiments.
  2. Routing correctness may be too restrictive. Fixed reference slots permit alternative agents but can reject valid alternative topological orders. An audit of existing baseline errors would clarify how much this affects the reported advantage.
  3. Extensibility and resource controls need clarification. Table 4 does not establish how often correct new-agent selection and valid completion occur jointly, while shared generation backends do not ensure comparable identity-reference support or retry budgets. These protocol questions are largely answerable from existing records.
Questions for authors
  1. How many independent raters participated, how were missing responses handled, and what uncertainty remains after accounting for both raters and cases?
  2. What do existing real-video logs show about task-stack edits, recovery and evaluator retries, and how were baseline generation budgets matched?
  3. Do baseline routing errors include executable alternative orders? For Table 4, what are the Selection/Pass denominators, execution conditions and joint-success counts?
9 comments
AuthorsOpenReview ↗

W1. Independent Contributions of Task Stack and Workspace Part 1

We thank the reviewer for the careful evaluation of FrameWorkers and the positive assessment of our system architecture and experimental results. To address the main concerns, we conduct additional controlled ablations of the Task Stack and Workspace, and further clarify details concerning Routing Evaluation, Extensibility, Human Evaluation, and generation resource settings.

W1. Independent Contributions of Task Stack and Workspace

The Task Stack and Workspace address two distinct problems: execution-state-aware dynamic task recovery and semantic asset resolution across Sub-agents, respectively. We validate their contributions through separate controlled experiments.

(1) Dynamic Task Stack

The Dynamic Task Stack improves the reliability of runtime failure recovery by explicitly maintaining execution states and modifying pending tasks. To validate this mechanism, we start from manually verified correct production chains and construct controlled perturbation cases by injecting three types of runtime failures: T1 (quality-check failure), T2 (one missing required upstream Producer), and T3 (two missing upstream Producers). These failures prevent the original production chains from completing normally and require the Replanner to restore the necessary production capabilities and their dependencies based on the failure information. We construct 1,556 and 466 perturbed cases on DCE and DCH, respectively.

To distinguish the effects of execution-history visibility from those of structured task updates, we compare three recovery strategies. All methods use the same Replanning LLM, user goal, Agent Descriptors, and failure report; only the execution-state information available to the Replanner and the way the new plan is applied differ:

MethodInformation Available to the ReplannerHow the New Plan Is Applied
Stateless Full RegenerationUser goal and failure report, without execution historyRegenerate the entire plan and execute from scratch
Log-aware Full RegenerationUser goal and complete execution log, including the initial plan, completed steps, and failure recordsRegenerate the entire plan and execute from scratch
FrameWorkersCompleted, failed, and pending steps in the Task StackModify the pending suffix while preserving completed artifacts

All methods are evaluated using the same recovery-correctness criterion: the recovered production chain must contain all required capabilities and satisfy Producer–Consumer dependencies.

ReplannerDatasetStatelessLog-awareFrameWorkers
Gemini 2.5 FlashDCE (1,556)88.4%98.3%99.0%
Gemini 2.5 FlashDCH (466)86.5%98.5%99.4%
Gemini 2.5 ProDCE (600)89.7%93.0%99.2%
Gemini 2.5 ProDCH (466)90.6%93.1%99.8%

FrameWorkers improves over Stateless Full Regeneration by 9.2–12.9pp across all four experimental settings, with all corresponding 95% CIs excluding zero. This demonstrates the importance of execution-history visibility and state management for reliable recovery.

Compared further with Log-aware Full Regeneration, FrameWorkers achieves gains of +0.7pp (95% CI [−0.1, 1.5]) and +0.9pp ([−0.4, 2.2]) with Gemini 2.5 Flash; neither difference is statistically significant. With Gemini 2.5 Pro, however, FrameWorkers achieves improvements of +6.2pp ([4.1, 8.4]) and +6.7pp ([3.7, 10.1]), respectively.

These results show that providing execution history already substantially improves recovery performance, while the Task Stack further organizes execution states into actionable task records, allowing the Replanner to modify pending tasks directly rather than regenerate the entire production plan. This structured recovery approach yields higher recovery correctness, particularly with Gemini 2.5 Pro.

AuthorsOpenReview ↗

W1. Independent Contributions of Task Stack and Workspace Part 2

(2) Execution-level Validation

FrameWorkers not only recovers correct production plans but also increases task completion after failures and reduces redundant execution by reusing existing results. The preceding experiments primarily evaluate capability coverage and dependency correctness in the recovered production chains. However, a correct plan does not necessarily imply successful execution, nor does it directly measure the execution overhead of different recovery strategies. We therefore conduct Execution-level Validation to assess the role of the Task Stack during actual task orchestration and execution.

Specifically, we construct 75 controlled failure cases spanning T1, T2, and T3, with 25 cases per failure type. All methods use the same initial production plans and failure conditions, and run with the same Director, Assistant, Workspace, and task execution framework. Only the post-failure recovery strategy is varied.

Fixed Pipeline stops upon failure; Stateless Regeneration regenerates and executes the entire plan in a new Workspace; and FrameWorkers preserves completed artifacts in the original Workspace, modifies the remaining tasks through Suffix Editing, and continues execution. The latter two strategies allow up to two rounds of Replanning.

We evaluate three metrics: Task Completion, whether all required production capabilities are completed in a valid dependency order; Agent Executions / Case, the total number of Agent executions per case; and Re-executions / Case, the number of repeated executions of previously completed tasks.

MethodCompletionAgent Executions / CaseRe-executions / Case
Fixed Pipeline0%1.630
Stateless Regeneration90.7%8.611.65
FrameWorkers100%6.570.05

FrameWorkers successfully completes the production tasks in all 75 cases, achieving 100% Completion, compared with 90.7% for Stateless Regeneration.

Meanwhile, its average Agent Executions decrease from 8.61 to 6.57, and Re-executions decrease from 1.65 to 0.05. This is because Stateless Regeneration must execute the regenerated plan from scratch, whereas FrameWorkers can directly reuse completed task results in the Workspace and execute only the remaining production steps after recovery.

Thus, the value of the Task Stack lies not only in the correctness of recovery plans but also in integrating dynamic recovery with the existing execution state, enabling the system to complete post-failure production tasks with less redundant work.

AuthorsOpenReview ↗

W1. Independent Contributions of Task Stack and Workspace Part 3

(3) Workspace Ablation

The Workspace's Semantic Asset Resolution more reliably selects the correct input assets for different Sub-agents, thereby improving the execution success rate of the entire production workflow. To isolate this mechanism, we compare four asset-selection strategies on the same 204 execution cases as in Table 3.

In FrameWorkers, each Sub-agent produces new media assets during execution and stores them in the Workspace. Subsequent Sub-agents must select assets that meet their input requirements from the available assets. For example, KeyFrameAgent needs reference images of the corresponding characters and scenes, rather than arbitrary assets of the same image type. Therefore, Asset Resolution selects the correct inputs from previously generated candidate assets based on the input requirements of the current Sub-agent.

In the experiment, each case uses a fixed, manually verified production chain. All methods execute the same Sub-agents and maintain the same Evaluator Rework Budget (the permitted number of evaluation and rework attempts), execution conditions, and task order. The only variable is the asset-selection strategy: how each Sub-agent selects its inputs from assets already stored in the Workspace before execution.

We compare the following four methods:

MethodAsset Selection Strategy
Standard Pipeline WiringUse 46 manually configured Producer–Consumer wiring rules to bind upstream Sub-agent outputs to downstream inputs.
Sequential PassingDirectly use the asset of the required media type generated in the preceding step.
Type-based MatchingSelect matching assets of the required media type from existing assets in generation order.
FrameWorkers WorkspaceUse an LLM to dynamically select the most appropriate asset based on the semantic description of the current input requirement and the Captions (text descriptions) of existing assets, without predefined wiring rules.

The Workspace selects inputs based only on the current Sub-agent's input requirements and the descriptions of existing assets, without needing to read the complete production plan. The other three strategies do not use asset Captions or additional LLM calls.

We use Execution Pass to measure execution success: a case passes only if all Sub-agents in the production chain obtain assets satisfying their input requirements and the entire execution workflow completes successfully.

MethodExecution Pass
Standard Pipeline Wiring45.6%
Sequential Passing9.3%
Type-based Matching23.0%
FrameWorkers Workspace99.0%

FrameWorkers reaches an Execution Pass of 99.0%, an improvement of 53.4pp over Standard Pipeline Wiring with 46 manually configured wiring rules.

Because all methods use the same production chains, Sub-agents, and execution conditions, the performance differences arise from how input assets are selected. These results validate the independent contribution of Semantic Asset Resolution: compared with fixed upstream–downstream wiring or matching solely by media type, semantic selection based on input requirements and asset descriptions more accurately handles cross-stage asset dependencies in complex production workflows.

Reviewer_TShLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FrameWorkers replaces fixed AIGC video pipelines with runtime orchestration. A Director (Qwen3-8B, LoRA, SFT then GRPO with an F1-coverage plus ordering reward) edits a Dynamic Task Stack through ADD/DELETE over a catalog of 20 sub-agents exposed only by natural-language descriptors; an Assistant executes each task against a persistent Workspace with semantic input retrieval and schema/LLM output checks. Evaluation covers five axes: routing accuracy on two template-generated test sets, recovery on 991 perturbed task stacks, execution robustness across LLM backends under mock materialization, two unseen sub-agents, and end-to-end video quality on 20+20 cases via an MLLM judge and a human study.

Critical strengths
  1. Coherent systems design: descriptor-only registration and the strict-out/loose-in Workspace are sensible, and the frozen-Director extensibility test, though small, shows descriptors are actually read.

  2. Broad evaluation footprint with candid reporting: mock materialization, rater-level significance tests, an author among raters, and the reconstructed Figure 2 are all disclosed.

  3. The ablation is informative: structured CoT is the weakest SFT format yet the best after GRPO (+33.5 DCH chain points), and 8B only beats 4B after RL.

Critical weaknesses
  1. The headline routing comparison is not like-for-like. Train and test chains come from the same template generator and eleven categories; the Director sees roughly 2.8k in-distribution examples while every baseline is zero-shot without in-context examples.

  2. The dynamic replanning that motivates the paper is thinly tested: recovery uses an off-the-shelf LLM rather than the trained Director; plan-level perturbations are near-trivial at 99.8%; there is no baseline such as replanning from scratch to show the Task Stack abstraction contributes; and there is no end-to-end ablation of the Task Stack or Workspace retrieval against a fixed pipeline.

  3. End-to-end evidence is weak where it matters. Against UniVA, the strongest prior system, the judge margin is 0.05 and human gains are non-significant at rater level. MovieAgent’s judge scores (C3 = 1.25, C6 = 1.15) suggest a poorly adapted baseline, and the adaptation is not described.

  4. The plan space is restricted to sequential chains with no repeats or branching, excluding per-shot loops and parallel generation typical of real production, so “dynamic task management” amounts mostly to failure-time edits of a linear list.

Questions for authors

As shown in weaknesses

7 comments
AuthorsOpenReview ↗

W1. Routing Fairness and Generalization

We thank the reviewer for the careful evaluation of FrameWorkers and the positive assessment of our system design and experimental results. In response to the concerns regarding Routing Fairness, Dynamic Replanning, End-to-end Evaluation, and Task Representation, we further clarify the relevant design choices and provide additional experiments and analyses.

W1. Routing Fairness and Generalization

First, we clarify that the Zero-shot Baselines in the original paper were provided with complete task knowledge and planning guidance. All models used the same core Planning Prompt, including Routing Rules, Sub-agent Descriptors (Purpose, Trigger, Inputs, Outputs, and dependencies), and requirements for capability selection and dependency ordering. Zero-shot means only that no completed planning examples were provided, not that the models lacked task knowledge.

Regarding data construction, Template-based Generation in the original paper refers to the construction of Ground-truth Task Chains, rather than the use of identical text templates to generate training and test instructions. Specifically, DCE shares task-chain structures with the training set, but its test instructions were constructed independently; DCH differs from the training set in both user instructions and task-chain structures, and is primarily intended to evaluate the Director's generalization to unseen Sub-agent combinations and their dependencies.

To further assess whether general-purpose LLMs can match the planning performance of the specifically trained Director when provided with planning examples, we conducted additional comparisons using Gemini 3.1 Pro and Gemini 2.5 Pro under 0-, 5-, 10-, and 20-shot settings. For each Few-shot setting, examples were randomly sampled from the training set, with three independently repeated experiments using three predetermined random seeds. We report the mean Chain Accuracy (%) over the three runs.

DatasetModel0-shot5-shot10-shot20-shot
DCEGemini 3.1 Pro78.280.382.091.4
DCEGemini 2.5 Pro79.785.685.889.7
DCHGemini 3.1 Pro51.056.055.563.0
DCHGemini 2.5 Pro52.051.055.557.0

Under the same evaluation protocol, the trained Qwen3-8B Director achieves 91.0% Chain Accuracy on DCE and 81.5% on DCH, without requiring Few-shot Demonstrations at inference time.

On DCE, 20-shot Gemini 3.1 Pro matches our Director, indicating that In-context Demonstrations can effectively improve planning performance on seen task structures. However, on DCH, the strongest Few-shot Baseline reaches only 63.0%, remaining 18.5pp below our 81.5%.

These results show that although Few-shot Prompting can improve the planning accuracy of general-purpose LLMs, a substantial gap remains on unseen task-chain structures. This further supports the dedicated training design of the Learned Director in FrameWorkers: Structured-CoT SFT and GRPO optimize production capability selection and dependency-aware planning, enabling the Director not only to organize seen task structures accurately but also to plan unseen Sub-agent combinations more reliably.

AuthorsOpenReview ↗

W2. Dynamic Replanning and Task Stack Part 1

W2. Dynamic Replanning and Task Stack

Replanning setting. Replanning is a relatively simpler setting because substantial information about the failure is already available. Consequently, both untrained general-purpose LLMs such as Gemini and our model can handle this setting. We therefore use Gemini 2.5 Flash as the Runtime Replanner and keep the Replanning LLM fixed in the controlled experiments to examine the contribution of the Task Stack.

Controlled Replanning. To independently validate the dynamic recovery mechanism, we start from manually verified correct production chains and construct controlled perturbation cases by injecting three types of runtime failures: T1 (quality-check failure), T2 (one missing required upstream Producer), and T3 (two missing upstream Producers, including partial or silent omissions). Each perturbed case represents the execution state of an original production chain under a particular failure condition and requires the Replanner to restore the necessary production capabilities and their dependencies based on the failure information. Following this procedure, we construct 1,556 and 466 perturbed cases on DCE and DCH, respectively.

All methods receive the same user goal, Agent Descriptors, and failure report. We compare three recovery strategies:

MethodReplanning Strategy
Stateless Full RegenerationGenerate an entirely new plan from the user goal and failure report without access to execution history, and execute it from scratch.
Log-aware Full RegenerationUse the complete execution log to generate an entirely new plan and execute it from scratch.
FrameWorkersUse the execution state in the Task Stack to modify the pending suffix through Suffix Editing, while preserving completed work.

All methods are evaluated using the same recovery-correctness criterion: the resulting task chain must contain all required capabilities and satisfy the Producer–Consumer execution dependencies.

ReplannerDatasetStatelessLog-awareFrameWorkers
Gemini 2.5 FlashDCE (1,556)88.4%98.3%99.0%
Gemini 2.5 FlashDCH (466)86.5%98.5%99.4%
Gemini 2.5 ProDCE (600)89.7%93.0%99.2%
Gemini 2.5 ProDCH (466)90.6%93.1%99.8%

The improvements of FrameWorkers over the Baselines and the corresponding paired-bootstrap 95% CIs clustered by source chain are as follows:

ReplannerDatasetvs Stateless (pp)vs Log-aware (pp)
Gemini 2.5 FlashDCE+10.6 [8.5, 12.8]+0.7 [−0.1, 1.5]
Gemini 2.5 FlashDCH+12.9 [8.7, 17.5]+0.9 [−0.4, 2.2]
Gemini 2.5 ProDCE+9.5 [6.4, 12.9]+6.2 [4.1, 8.4]
Gemini 2.5 ProDCH+9.2 [5.4, 13.5]+6.7 [3.7, 10.1]

Across both models and both datasets, FrameWorkers outperforms Stateless Full Regeneration by 9.2–12.9pp, demonstrating the importance of execution-state visibility for reliable recovery.

Log-aware Full Regeneration performs similarly to FrameWorkers with Flash, but is 6.2pp and 6.7pp lower with Pro, respectively. These results further support the combined value of the structured Task Stack and Suffix Editing: they allow the system to preserve completed work and reduce the risk of changing previous decisions when regenerating an entire plan.

AuthorsOpenReview ↗

W2. Dynamic Replanning and Task Stack Part 2

Execution-level Validation. The preceding Controlled Replanning experiments primarily evaluate the planning correctness of the recovered task chains. To further validate the effects of these recovery strategies during actual task orchestration and execution, we construct 75 controlled failure cases, covering T1 (quality-check failure), T2 (one missing required upstream Producer), and T3 (two missing upstream Producers), with 25 cases per type.

The experiments start from the same initial production plan, execute Sub-agents in dependency order, and inject predetermined failures during execution. After a failure occurs, we compare three handling strategies:

  • Fixed Pipeline: Stop execution upon failure, without dynamic recovery.
  • Stateless Regeneration: Regenerate a complete production plan from the user goal and failure information, and execute all tasks from scratch.
  • FrameWorkers: Maintain the states of completed, failed, and pending tasks in the Task Stack, repair the remaining plan through Suffix Editing, and continue execution by reusing existing production results in the Workspace.

All methods face the same initial plan and failure conditions, and the latter two strategies allow up to two rounds of Replanning. We record three metrics: Task Completion, whether all required production capabilities are completed in a valid dependency order; Agent Executions / Case, the total number of Agent executions per case; and Re-executions / Case, the number of repeated executions of previously completed production steps.

MethodCompletionAgent Executions / CaseRe-executions / Case
Fixed Pipeline0%1.630
Stateless Regeneration90.7%8.611.65
FrameWorkers100%6.570.05

FrameWorkers successfully completes failure recovery and subsequent task execution in all 75 cases, achieving 100% Completion, compared with 90.7% for Stateless Regeneration. Meanwhile, average Agent Executions decrease from 8.61 to 6.57, and Re-executions decrease from 1.65 to 0.05.

These results show that the Task Stack not only helps the Replanner generate correct recovery plans but also integrates those plans with the existing execution state, improving task completion while avoiding redundant execution of completed production steps.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Invitation to Further Discussion

Dear Reviewers,

Thank you for your time and valuable feedback. We provided point-by-point responses a few days ago and hope they address your concerns. With only a few days left in the discussion period, we would appreciate it if you could review our responses and share any remaining concerns, so that we still have time to address them.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Revised Manuscript Uploaded — Summary of Changes and Request for Feedback

Dear Reviewers,

We have uploaded our revised manuscript, with all changes highlighted in red and additional experiments included in Appendix I. We have addressed the concerns raised in the reviews and listed the exact locations of the corresponding revisions below for your convenience.

Reviewer TShL

W1 (routing fairness): Section 3 and Appendix I.2 (Table 10).

W2 (replanning and Task Stack): Table 3 and Appendix I.3.

W3 (end-to-end evaluation): Appendix I.6–I.7 (Tables 17–18).

W4 (sequential tasks): Section 2.

Reviewer 3pQz

W1 (Task Stack and Workspace): Table 3 and Appendix I.3–I.4 (Table 14).

W2 / Q3 (alternative orders): Section 3 and Appendix I.1.

W3 / Q3 (extensibility): Table 5 and Appendix I.5.

Q1 (human evaluation): Appendix I.6 (Table 16).

Q2 (runtime behavior and budgets): Appendix I.3 and I.7.

Reviewer ExCC

W1 (novelty): Appendix I.9 (Table 20), Table 3, and Appendix I.4.

W2 (video comparisons): Video links at the beginning of the Appendix.

W3 / Q1 (cross-shot consistency): Appendix I.8 (Table 19).

Note: Due to the addition of a new table in the main text, the original Tables 3–5 are now Tables 4–6.

We would greatly appreciate it if you could take a look at the revised manuscript and let us know whether our revisions have addressed your concerns.

Thank you again for your time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.