acceptodds
Withdrawn from ICLR 2027

SkillPE: Creativity-Oriented Cinematic Skill Evolution for Text-to-Video Prompt Engineering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Achieving high-quality, cinematic results in text-to-video generation remains challenging for non-experts, whose prompts often lack professional narrative and creative design. We propose SkillPE, a prompt engineering (PE) framework that evolves reusable cinematic skills from expert-authored seeds. SkillPE represents shot logic, composition, lighting, sound design, and other filmmaking cues in a fine-grained format, and retrieves movie references categorized as resonators (good matches), dissonants (weak matches), and divergents (creatively useful near-misses). The first two refine when and how a skill should be applied, while divergents inspire alternative cinematic realizations at different degrees of modification while preserving the user intent. Candidate skills are assessed through generated videos along prompt fidelity, cinematic quality, narrative appeal, and creativity to construct the final skill libraries. Experiments on StoryEval and VBench show improvements of up to 1.40 points over the strongest external baseline and 0.51 points over seed skills on 7-point four-dimensional evaluation, while remaining competitive on benchmark-native metrics. Overall, SkillPE offers a practical approach to balancing fidelity and creativity in cinematic text-to-video generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generative Video Model, AI Agent, Skills

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JDjoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In my understanding, this paper introduces SkillPE, a prompt-engineering framework for improving the cinematic quality of text-to-video generation. It starts with expert-designed cinematic skills covering shot structure, camera movement, composition, lighting, editing rhythm, and sound design. The framework then uses real film clips as positive examples, negative examples, and creative near-misses to refine the skills and generate alternative versions. Each candidate skill is applied to prompts to generate videos, which are evaluated for prompt fidelity, cinematic quality, narrative appeal, and creativity. The best-performing skills are selected to form the final skill library. For the results, experiments show that SkillPE improves cinematic quality, storytelling, and creativity over conventional prompt rewriting methods. In essence, however, the method is mainly an expert rule library combined with reference-guided rule refinement and video-based candidate selection, rather than a truly autonomous or end-to-end self-evolving system.

Critical strengths

The paper addresses the real difficulty non-experts face when creating cinematic text-to-video prompts.

The paper evaluates multiple baselines, two video-generation backbones, benchmark metrics, custom metrics, and a human study.

Critical weaknesses

This method provides a rule database based on experts and a large amount of data. I think this is a good open-source contribution, but it is not a research work and does not provide me with any new inspiration. I am not sure whether such works are acceptable for ICLR. So could the author state the reusable design or findings not discovered before clearly?

The Seed Skill already provides most of the improvement, so whether the subsequent evolution is necessary is questionable. This also shows that the largest improvement comes from manual design, and fails to demonstrate that this framework can generalize to any prompt-engineering scenario.

Questions for authors
  1. Could the authors evaluate the method on prompt-engineering tasks in other domains, such as advertisement generation? Otherwise, the claim that this is a general skill-evolution framework is not sufficiently supported.

  2. How would the method perform if the initial Seed Skills were generated by an LLM rather than manually designed by experts? I am curious whether the knowledge from the experts is already learned by some SOTA LLM.

  3. How does the size of the skill library affect the final performance? Would having too many final skills introduce redundancy or reduce the efficiency of skill selection and application? What happens if there are only five or seven Seed Skills from experts?

Reviewer_LgrHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a cinematic skill library for text-to-video prompt rewriting. Expert-written skills are refined using movie references categorized as resonators, dissonants, and divergents. The resulting candidates are evaluated on development prompts, and a library is selected for each backbone. During inference, an LLM selects a skill and rewrites the input prompt. The experiments show that the seed library accounts for much of the improvement in cinematic and creative quality, with further gains from evolution and selection.

Critical strengths

The experimental design addresses several factors that could explain the improvement. Comparisons at matched prompt lengths show that the benefit is not simply obtained by generating longer prompts. The seed library is evaluated separately from the evolved libraries, and assessed selection is compared with random candidate selection. These experiments provide evidence for the contribution of structured cinematic guidance while avoiding attribution of the entire improvement to evolution.

Critical weaknesses
  1. The experiments support skill evolution and candidate selection, but the contribution of the movie references themselves is not isolated. Table 3 only varies which reference-guided branch is used, resonators and dissonants are always pooled (App. E.3), and the refinement branch alone barely moves StoryEval. There is no control that mutates the seed without any references under the same mutation budget (App. E.4.1) and the same selection procedure, so the benefit of references over plain LLM mutation is not established.
  2. On LTX-2.5, the Gemini fidelity score drops after evolution (Table 1) while the human fidelity score rises (Table 2). App. G.2 gives an aggregate breakdown and Fig. 4 shows one omitted-event case, but there is no per-prompt analysis of whether the rewrite drops events or alters subjects and their relations, so it remains unclear when cinematic rewriting preserves the original intent.
Questions for authors
  1. Does category-based guidance improve over using the same reference pool without categorization, under matched candidate and selection budgets?
  2. For the hard StoryEval subset, can you tally per prompt whether the rewrite drops an event or alters a subject or relation, and report human fidelity on that subset separately? The human study prompts are balanced across difficulty (Sec. 4.2), so the hard cases may be masked.
Reviewer_BGJcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SkillPE, a framework that improves text-to-video prompt engineering through reusable cinematic skills. It uses movie clips as references to refine existing skills and explore new variants with different levels of creativity. It then selects the final skills based on generated video quality, considering prompt fidelity, cinematic quality, narrative appeal, and creativity.

Critical strengths
  1. The paper moves beyond simple prompt rewriting by using structured cinematic skills to guide video generation. The combination of skill refinement and creative exploration is well motivated.
  2. The authors test the method on two video generation models and two benchmarks, and conduct ablation studies and a human evaluation, providing relatively diverse evidence for the effectiveness of the proposed approach.
Critical weaknesses
  1. The ablation study does not separately examine the roles of resonators and dissonants in skill evolution. In particular, it remains unclear whether dissonants help identify the applicability boundaries of each skill.
  2. The reported gains reflect both skill evolution and subsequent skill selection, making it difficult to reveal how much improvement comes from the evolution process itself. The authors should report results for evolved skills without the selection stage.
  3. The paper does not compare SkillPE with relevant existing approaches to prompt evolution or agent skill refinement. For example, comparisons with iterative LLM-based refinement or mutation-based candidate generation would help verify whether the proposed reference-guided evolution provides benefits beyond these simpler alternatives.
  4. Gemini 3.1 Pro is used to evaluate the generated videos and select the final skills. The reliability of its scores, especially for subjective aspects such as creativity and narrative appeal, remains unclear. Since the same model is used for both selection and evaluation, independent evaluation and stronger validation against human judgments would make the results more convincing.
  5. The appendix reports a 1.30–2.12 point decrease in the overall VBench score after skill evolution on MiniMax-H3, with larger drops in semantic alignment than in video quality. This raises the question of whether the improved cinematic quality and creativity come at the cost of scene, object, or spatial consistency. The authors should better explain this trade-off and clarify the extent to which SkillPE improves video generation quality beyond its custom evaluation metrics.
Questions for authors

See the "Critical Weaknesses".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.