acceptodds
Under review as a conference paper at ICLR 2027

ReKey: Benchmarking and Learning Agentic Keyframe Reflection for Video Generation

20 views

Abstract

Recent video generation models have advanced substantially in visual fidelity and motion modeling, yet generated videos still suffer from semantic and temporal inconsistencies. This limitation has motivated Reflective Video Generation (RVG), a paradigm that introduces explicit reflection and correction into the generation process. We introduce ReKey-Bench to evaluate RVG. It comprises 400 video cases and 784 keyframe evaluation samples, covering keyframe verification, rectification, and downstream video improvement. Evaluation of existing vision-language models (VLMs) on ReKey-Bench reveals that they may overlook visual errors or produce rectifications that violate the keyframe plan. To address these limitations, we propose ReKey, an agent framework for evidence-grounded keyframe reflection. We train its reflection agent through on-policy self-distillation, providing a frozen teacher with question-answer-evidence (QAE) as privileged context. ReKey achieves gains of 13.3 points in keyframe verification F1 and 15.9 points in keyframe rectification score over Qwen3.5 on ReKey-Bench, while also improving downstream video generation and achieving consistent gains on T2V-CompBench and VBench-2.0. Our code and models will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video generation, multimodal agents, keyframe reflection, on-policy self-distillation, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FCCeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ReKey-Bench, a benchmark for evaluating keyframe verification, keyframe rectification, and downstream video generation in reflective video generation. The benchmark consists of 400 video cases and 784 keyframe samples. In addition, the paper proposes ReKey, which trains a keyframe reflection agent via on-policy self-distillation. A frozen teacher is provided with question-answer-evidence (QAE) as privileged context, while the student learns to perform keyframe verification and rectification using only the original inputs. Experiments based on the Qwen3.5 series show that ReKey outperforms multiple training and inference baselines in keyframe verification, rectification, and downstream video generation. The paper further evaluates ReKey on T2V-CompBench and VBench-2.0, and conducts ablation studies with different forms of teacher guidance to validate the effectiveness of QAE supervision.

Critical strengths

None

Critical weaknesses

The paper proposes ReKey-Bench and ReKey. ReKey-Bench consists of 400 video cases and 784 keyframe samples, and evaluates reflective video generation at three stages: keyframe verification, keyframe rectification, and downstream video improvement. ReKey, meanwhile, employs on-policy self-distillation with QAE as privileged context, enabling the model to perform more reliable keyframe reflection and correction at inference time without requiring additional evidence inputs. Experiments show that ReKey outperforms multiple baselines in keyframe verification, rectification, and downstream video generation, with further support from QAE ablations, bootstrap confidence intervals, and evaluations using different evaluators. However, the methodological novelty is relatively limited, as it mainly builds on existing OPSD and privileged-evidence distillation paradigms. In addition, on external T2V benchmarks, ReKey uses keyframe-conditioned generation while some comparison methods rely on different generation interfaces, making it difficult to attribute the observed gains solely to ReKey itself. Furthermore, the benchmark depends on a fixed image editor, video generator, and VLM evaluator, and its generalization across different backbones remains to be further validated.

Questions for authors
  1. Fairness of the external T2V comparison. ReKey uses keyframe-conditioned video generation based on Ref2VA, while some baselines use FL2VA. Could the authors provide comparisons under the same generation pipeline to more accurately isolate the contribution of ReKey itself?

  2. Generalization across backends. ReKey-Bench relies on fixed image editors, video generators, and evaluation models. Could the authors clarify whether ReKey maintains its relative advantage when using different image editors or video generators?

  3. Novelty relative to existing reflection and OPSD methods. Given that R³-Bench has already studied reflection/rectification in visual generation, and recent works such as Vision-OPD and Video-OPSD also employ on-policy self-distillation with privileged visual evidence, what is the core methodological novelty of ReKey beyond applying these ideas to sequential keyframe-based video generation?

6 comments

Response to Q1 (External Comparison Fairness):

Thank you for your comments. We clarify the original backend selection and provide Ref2VA baseline and reflection-agent comparisons. We have added a summary of these comparisons to Sec. 4.2, with the backend clarification, experimental details, and results provided in Appendix C.5 (Tables 13 and 14) of the updated submission. The changes are highlighted in blue\color{blue}{\text{blue}}.

Original Setup. In the original experiments, configurations with reference keyframes use Ref2VA, while configurations without reference images use FL2VA in text-only t2va mode. This follows the documented H3 interfaces, where Ref2VA requires a reference input and FL2VA supports text-only generation.

Nevertheless, we agree that comparisons under a shared generation pipeline provide a more direct assessment of ReKey’s contribution. We therefore provide two complementary comparisons.

Ref2VA Baseline Comparison. We compare ReKey with six baselines under the same two-stage Ref2VA generation procedure, using one reference image for coarse video generation and three keyframes for final video generation. The baselines share the same unedited candidate keyframes, while ReKey verifies and corrects its candidates. Each optimization method retains its own procedure and obtains any required video feedback from Ref2VA-generated videos. As shown in Tab. R1, ReKey achieves the highest average score on both benchmarks when the baselines also use Ref2VA.

Tab. R1. Comparison with six baselines using Ref2VA on all cases in the selected benchmark dimensions. ReKey (w/o reflection) omits keyframe verification and rectification. Scores follow the original benchmark aggregation over each method’s valid results and are reported as percentages. Confidence intervals are for paired differences computed as ReKey minus each comparator on cases with valid scores for both methods, expressed in percentage points.

MethodT2V-CompBench ↑95% CIVBench-2.0 ↑95% CI
ReKey (w/o reflection)65.94[+0.38, +2.80]58.35[+2.41, +8.69]
SCMAPR65.10[+1.06, +3.83]59.07[+1.37, +8.30]
SSPO64.71[+1.58, +4.09]59.53[+0.79, +7.94]
PhyT2V65.82[+0.48, +2.96]57.77[+2.96, +9.30]
NEWTON64.58[+1.66, +4.27]60.00[+0.56, +7.15]
VPO65.57[+0.60, +3.31]61.17[−1.05, +6.47]
ReKey67.54—63.89—

Reflection Agent Comparison. We compare Base Qwen3.5-9B, SFT, GRPO, Label-guided OPSD, and ReKey within the same generation pipeline. All agents share the planning procedure, initial assets, image editor, video generator, generation settings, and reflection and editing budgets. We evaluate a fixed subset of 250 T2V-CompBench cases and 300 VBench-2.0 cases, with 50 cases per dimension. Scores follow the original benchmark aggregation over each method’s valid results. We estimate paired 95% confidence intervals for ReKey’s improvement over each comparator using cases with valid scores for both methods.

Tab. R2. Reflection agent results on the fixed 550-case subset. Scores and confidence intervals follow the convention in Tab. R1.

Reflection agentT2V-CompBench ↑95% CIVBench-2.0 ↑95% CI
Base66.56[+0.95, +6.70]66.36[−0.01, +8.78]
SFT66.96[+0.44, +6.40]68.56[−2.01, +6.76]
GRPO67.09[+0.60, +6.19]66.21[+0.37, +8.41]
Label-guided OPSD66.52[+1.08, +6.61]67.00[−0.67, +7.33]
ReKey69.93—70.33—

Response to Q2 (Backend Generalization):

Thank you for your comments. We evaluate an alternative image editor, two video generators, and two additional evaluators. We have summarized these results in Section 4.1 and provided the editor and generator comparisons in Appendix C.6 (Tables 16 and 17) and the evaluator comparison in Appendix C.3 (Table 9) of the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Alternative Image Editor. We evaluate Qwen-Image2.1 with prompt enhancement (PE) on 784 independent keyframe tasks and 400 sequential video cases per method. All reflection agents share the same editing configuration. As shown in Tab. R3, ReKey achieves the highest keyframe rectification and video improvement scores with this configuration. The editor and generator comparisons use Qwen3.5-9B reflection agents and Qwen3.8-27B for evaluation.

Tab. R3. Results with Qwen-Image2.1 + PE on ReKey-Bench.

Reflection agentSrectI\mathcal{S}_{\mathrm{rect}}^{I} ↑SimpV\mathcal{S}_{\mathrm{imp}}^{V} ↑
Base0.31680.5198
SFT0.36490.5631
GRPO0.34590.5549
Label-guided OPSD0.30190.5266
ReKey0.41930.5863

Alternative Video Generator. We evaluate LTX-2.3 on all 400 ReKey-Bench cases and Wan2.1-VACE on a fixed 100-case subset. For each generator, video improvement is measured relative to its own text-only Direct baseline on the same cases. ReKey achieves the highest video improvement score with both generators (Tab. R4).

Tab. R4. Video improvement scores SimpV\mathcal{S}_{\mathrm{imp}}^{V} with alternative generators. U3 uses the original three keyframes without reflection or editing. Reflection agents use Qwen3.5-9B.

MethodLTX-2.3 ↑Wan2.1-VACE ↑
Direct (text-only)0.00000.0000
U3 (original keyframes)0.55490.2948
SFT0.59330.3361
GRPO0.60750.3401
Label-guided OPSD0.56040.2975
ReKey0.63860.3603

Additional Evaluator. We rescore the original benchmark outputs with GLM-5.3-Flash and DeepSeek V4.1 Flash, covering 784 independent keyframe tasks and 400 video cases. ReKey retains the highest scores on both metrics under both evaluators (Tab. R5).

Tab. R5. Scores under additional evaluators. Each cell reports SrectI\mathcal{S}_{\mathrm{rect}}^{I} / SimpV\mathcal{S}_{\mathrm{imp}}^{V}.

Reflection agentGLM-5.3-FlashDeepSeek V4.1 Flash
Base0.341 / 0.5850.298 / 0.568
SFT0.478 / 0.5550.356 / 0.547
GRPO0.498 / 0.5920.322 / 0.559
Label-guided OPSD0.377 / 0.5680.238 / 0.549
ReKey0.552 / 0.6000.393 / 0.611

Response to Q3 (Distinction from Prior Work):

Thank you for your comments. We clarify the role of atomic QAE guidance and its distinction from prior reflection and on-policy distillation methods. We have clarified QAE's role in Section 3, expanded the analysis of teacher guidance under the same OPSD procedure in Section 4.3, and added a comparison with prior work in Appendix A (Table 5) of the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Our contributions. We introduce a diagnostic benchmark and an evidence-grounded approach to keyframe reflection. We study whether an agent can identify semantic and temporal errors, correct them while preserving satisfied requirements, and improve the resulting video. ReKey-Bench connects these questions through keyframe verification, rectification, and downstream video evaluation. The original-keyframe baseline, stage-level results, and analysis of corrected versus newly violated requirements help distinguish successful reflection from unnecessary or ineffective editing.

Distinction from prior work. R³-Refiner [1] learns visual verification and rectification through GRPO with verification and edited-image rewards. Vision-OPD [2] and Video-OPSD [3] use privileged crops and annotated evidence frames, respectively, to improve visual question answering. ReKey builds on on-policy distillation with atomic QAE guidance for corrective responses. Each QAE entry connects a visual requirement, its satisfaction status, and supporting image evidence. Together, these entries specify what needs correction and what should be preserved, including planned changes relative to the preceding keyframe. The teacher uses this guidance to supervise the student's verdict, explanation, and editing instruction, without requiring QAE at inference. The selected keyframe then provides the visual context for the next stage. Tab. R6 summarizes these distinctions and the corresponding evaluation components.

Tab. R6. Comparison with prior work. Locations refer to this paper.

FeatureR³-Bench [1]Vision-OPD [2]Video-OPSD [3]ReKey
Verification and rectification✓——✓ (Sec. 2.1)
Privileged on-policy distillation—✓✓✓ (Sec. 3, Fig. 4)
Plan-conditioned keyframe reflection———✓ (Sec. 2.1)
Atomic QAE teacher guidance———✓ (Sec. 3, Fig. 4)
Keyframe-to-video evaluation———✓ (Sec. 2.3, Fig. 2, Tab. 1)

OPSD Alone Does Not Explain the Gains. As shown in Table 4 of the original main paper, reproduced below as Tab. R7, we hold the backbone, training data, budget, and OPSD procedure fixed while varying only the teacher guidance. OPSD with verification labels, atomic QA, or length-matched QA+ performs below QAE on all four metrics. Compared with QA+, QAE improves F1 from 0.628 to 0.733, keyframe rectification from 0.367 to 0.440, and downstream video improvement from 0.378 to 0.438. These comparisons show that adopting OPSD with label or QA guidance does not reproduce the gains from QAE.

Tab. R7. Effects of teacher guidance on ReKey-Bench, reproduced from Table 4 of the original main paper. All variants use the same OPSD procedure. QA+ matches QAE's teacher-prompt length by repeating atomic QA.

Teacher guidanceAcc. ↑F1 ↑SrectI\mathcal{S}_{\mathrm{rect}}^{I} ↑SimpV\mathcal{S}_{\mathrm{imp}}^{V} ↑
Verification label0.7190.5450.2740.367
Atomic QA0.7950.6490.3560.359
QA+ (length-matched)0.7810.6280.3670.378
Atomic QAE (ReKey)0.8320.7330.4400.438

[1] Junjie Wang et al. Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generation. arXiv:2605.19639, 2026.

[2] Qianhao Yuan et al. Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation. arXiv:2605.18740, 2026.

[3] Ziyue Wang et al. Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models. arXiv:2608.27065, 2026.

Thank you for this suggestion. We have clarified these points in Sections 3 and 4.3 and Appendix A of the updated submission.

Reviewer_UMNcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Reflective Video Generation (RVG), where VLM agents verify and refine intermediate keyframes before video generation. The authors identify that existing agents often fail to detect visual errors or generate corrections inconsistent with the intended keyframe plan. The paper introduces ReKey-Bench, a benchmark for evaluating keyframe verification, rectification, and downstream video improvement. It further proposes ReKey, an evidence-grounded keyframe reflection agent trained with QAE-guided on-policy self-distillation, where a teacher receives privileged visual evidence supervision and transfers this capability to the student model. Experiments show that ReKey improves keyframe reflection performance and enhances downstream video generation quality on multiple benchmarks.

Critical strengths
  1. The paper addresses an important and underexplored problem in agentic video generation. The work focuses on enabling agents to verify and correct intermediate keyframes, and introduces ReKey-Bench to evaluate keyframe verification, rectification, and downstream video improvement, filling a gap in existing video generation benchmarks.
  2. The proposed evidence-grounded training strategy effectively improves reflection ability. The QAE-guided on-policy self-distillation transfers visual evidence supervision from a teacher model to the student without additional inference-time requirements, providing a practical approach for training reflection agents.
  3. The experimental evaluation is comprehensive and supports the effectiveness of the method. The paper compares ReKey with various VLM baselines and training strategies, and further validates improvements on downstream video generation benchmarks. Ablation studies also analyze the contribution of different teacher guidance designs.
Critical weaknesses
  1. The generalization of the proposed framework is insufficiently validated. The experiments mainly focus on Qwen-based VLM agents and a specific reflective video generation pipeline. It remains unclear whether the proposed approach can generalize to different VLM architectures, video generators, or alternative agent frameworks.
  2. The contribution of QAE supervision is not fully disentangled from additional structured supervision. Although the ablation studies compare different teacher guidance strategies, it is still unclear whether the improvement mainly comes from visual evidence grounding or simply from providing more detailed task-specific supervision.
  3. The benchmark may not fully isolate reflection reasoning from visual understanding ability. ReKey-Bench evaluates whether agents can identify keyframe violations, but some improvements may come from stronger visual recognition rather than better reflection or planning capabilities. More analysis separating these factors would strengthen the main claims.
Questions for authors
  1. The current evaluation mainly focuses on Qwen-based VLM agents within a specific RVG pipeline. Could the authors provide further evidence or discussion on whether ReKey generalizes to other VLM architectures, video generators, or alternative agent-based video generation settings?
  2. The ablation study compares different forms of teacher guidance, but the specific role of visual evidence in QAE supervision remains unclear. Could the authors analyze whether the performance gains primarily come from evidence grounding or from the additional structured supervision introduced by QAE?
  3. ReKey-Bench evaluates an agent’s ability to identify keyframe violations, but this capability may involve both visual understanding and higher-level reflection reasoning. Could the authors provide analysis separating improvements in visual error recognition from improvements in planning consistency and reflective decision-making?
6 comments

Response to W1 & Q1 (VLM and Video Generator Generalization):

Thank you for your comments. We evaluate an InternVL3.5-8B reflection backbone with Qwen-Image2.1 + PE and test two additional video generators. We have summarized these evaluations in Sec. 4.1 and provided the experimental details and results in Appendix C.6 (Tables 15 and 17) of the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Alternative Reflection Backbone. We extend the comparison to InternVL3.5-8B reflection agents using Qwen-Image2.1 with prompt enhancement (PE) as the image editor. All InternVL variants share the same editing configuration and MiniMax H3 video generator. As shown in Tab. R1, ReKey achieves the highest scores across all four metrics, supporting the effectiveness of QAE-guided training with this alternative reflection backbone and editing configuration.

Tab. R1. InternVL3.5-8B reflection agents with Qwen-Image2.1 + PE on ReKey-Bench.

Reflection agentAcc. ↑F1 ↑SrectI\mathcal{S}_{\mathrm{rect}}^{I} ↑SimpV\mathcal{S}_{\mathrm{imp}}^{V} ↑
Base0.68240.40000.17310.4913
SFT0.70030.50110.29000.5529
GRPO0.66200.52760.30890.5385
Label-guided OPSD0.69130.40390.12880.5272
ReKey0.70540.54970.33200.5724

Alternative Video Generator. We evaluate LTX-2.3 on all ReKey-Bench cases and Wan2.1-VACE on a fixed 100-case subset. As shown in Tab. R2, ReKey achieves the highest video improvement score with both generators, indicating that its benefits extend beyond MiniMax H3.

Tab. R2. Video improvement scores SimpV\mathcal{S}_{\mathrm{imp}}^{V} with alternative generators. U3 denotes generation from the original, unedited keyframes. Reflection agents use Qwen3.5-9B.

MethodLTX-2.3 ↑Wan2.1-VACE ↑
Direct (text-only)0.00000.0000
U3 (original keyframes)0.55490.2948
SFT0.59330.3361
GRPO0.60750.3401
Label-guided OPSD0.56040.2975
ReKey0.63860.3603

These results extend the evidence beyond Qwen-based agents and the original video generation backend.

Response to W2 & Q2 (Contribution of Visual Evidence):

Thank you for your comments. We explain how the teacher guidance ablations and visual-sensitivity analysis assess the contribution of visual evidence beyond structured QA supervision. We have expanded this discussion in Sec. 4.3 of the updated submission, with supporting analyses in Appendix C.1 (Fig. 10) and Appendix C.2 (Table 7). The changes are highlighted in blue\color{blue}{\text{blue}}.

Teacher Guidance Comparison. Atomic QA already provides structured supervision through individual visual requirements and their answers. QAE adds visual evidence describing the image content that supports each answer. The ablations in Sec. 4.3 use the same backbone, training data, training budget, and OPSD procedure. QA+ additionally controls prompt length by repeating QA content to match QAE within two text tokens (Appendix F.2).

Tab. R3. Effects of teacher guidance on ReKey-Bench, reproduced from Tab. 4 of the paper.

Teacher guidanceAcc. ↑F1 ↑SrectI\mathcal{S}_{\mathrm{rect}}^{I} ↑SimpV\mathcal{S}_{\mathrm{imp}}^{V} ↑
Verification label0.7190.5450.2740.367
Atomic QA0.7950.6490.3560.359
QA+ (length-matched)0.7810.6280.3670.378
Atomic QAE (ReKey)0.8320.7330.4400.438

Compared with QA+, QAE improves keyframe rectification by 0.073 (95% CI: [0.0002, 0.1457]) and video improvement by 0.060 (95% CI: [0.0137, 0.1068]). These paired confidence intervals use 10,000 bootstrap resamples over the 400 cases (Appendix C.2). The results support the contribution of visual evidence beyond atomic QA supervision and increased prompt length.

Visual Sensitivity. We further examine how the reflection agents respond to changes in visual input (Appendix C.1, Fig. 10). Holding the task text, preceding reference image, and response prefix fixed, we replace a valid target keyframe with its invalid counterpart. The mean increase in RECT probability, normalized over KEEP and RECT, is 0.417 for QAE and 0.326 for QA. Neither model receives privileged guidance during this evaluation. This greater separation between valid and invalid inputs supports improved sensitivity to visual violations after QAE-guided training.

Response to W3 & Q3 (Visual Understanding and Reflection):

Thank you for your comments. We connect the benchmark metrics and correction outcomes to an additional evaluation of explanations and correction instructions. We have added the scoring criteria and results on 262 human-labeled RECT keyframes to Appendix C.1 (Table 6) of the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Benchmark Design. ReKey-Bench evaluates reflective decisions and their effectiveness in satisfying the keyframe plan. Acc. and F1 assess whether KEEP/RECT decisions agree with human-verified labels. The keyframe rectification score measures whether the selected keyframes better satisfy atomic requirements for visual content, planned states, and changes relative to the preceding reference. With the editor fixed, this evaluates plan consistency through the outcomes of the agent’s decisions and correction instructions (Sec. 2.3). With the same Qwen3.5-9B backbone, ReKey improves accuracy from 75.9% to 83.2%, F1 from 60.0% to 73.3%, and keyframe rectification from 0.281 to 0.440 (Tab. 1).

Correction Outcomes. Appendix C.1 further analyzes the effects of the agents’ decisions and correction instructions through the same editor. Using the question weights underlying the main alignment scores, ReKey corrects 50.86% of initially unmet requirements, compared with 38.19% for Base, while introducing fewer violations of initially satisfied requirements, at 1.24% versus 1.82%. These results show that ReKey makes more effective corrections while better preserving valid content.

Reflection Output Quality. To further examine explanation grounding and planning consistency independently of editor execution, we additionally evaluate the agents’ original explanations and correction instructions. A fixed Qwen3.8-27B evaluator receives the task inputs and agent outputs, with method identities hidden. Reason assesses whether explanations are grounded in visible evidence and task requirements. Edit assesses whether instructions specify concrete corrections consistent with the keyframe plan and temporal continuity.

Tab. R4. Reflection output quality on all 262 human-labeled RECT keyframes.

Qwen3.5-9B methodReason pass rate ↑Edit pass rate ↑
Base88.17%51.15%
SFT89.69%59.92%
GRPO90.07%64.12%
Label-guided OPSD84.35%48.85%
ReKey93.51%67.94%

ReKey improves the Reason and Edit pass rates over Base by 5.34 and 16.79 percentage points, respectively. These evaluations complement verification accuracy by assessing whether explanations are grounded in visual evidence and correction instructions are consistent with the keyframe plan. Together with the correction outcome analysis, the results show improvements in both error detection and the quality and effectiveness of corrective responses.

Thank you again for your constructive comments and suggestions.

Reviewer_3eTqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies reflective video generation through sequential keyframe verification and rectification. It introduces ReKey-Bench, containing 400 video cases and 784 keyframe samples, with evaluations of verification accuracy, keyframe correction, and downstream video alignment. The proposed ReKey agent is trained through on-policy self-distillation. A frozen teacher receives privileged question–answer–evidence (QAE) annotations, while the student receives only the original task inputs. Token-level Jensen–Shannon divergence transfers the teacher’s predictions along student-generated response prefixes.

Critical strengths
  • A useful benchmark structure for diagnosing reflective generation. ReKey-Bench connects error detection, editing outcomes, and downstream video alignment instead of evaluating only final videos. The original-keyframe baseline, stage-level breakdowns, and analysis of corrected versus newly violated requirements help distinguish successful reflection from unnecessary or ineffective editing. This diagnostic structure is a meaningful contribution for developing keyframe-based generation systems.

  • Controlled evidence for the value of QAE supervision. The paper compares QAE-guided distillation with SFT, GRPO, label-guided distillation, atomic QA, and length-matched QA+. ReKey improves downstream video scores over these alternatives, with positive paired case-bootstrap intervals, and improvements over the base model extend to the 27B model. These experiments support the practical usefulness of privileged visual-evidence supervision, even though the underlying distillation method builds on existing OPSD.

Critical weaknesses
  • The external video-generation comparisons do not isolate learned reflection from changes to the generation pipeline. Appendix D.2 specifies Ref2VA for keyframe-conditioned generation and FL2VA for Direct and prompt-optimization baselines. ReKey additionally uses planning, initial video generation, and keyframe editing. Consequently, Tables 2–3 compare systems with different conditioning pathways and operations. They omit the most informative controls: the identical ReKey pipeline using the untrained, SFT, or GRPO reflection agent.

  • The new benchmark’s output-scoring validity remains insufficiently established. Both Gemini 3.1 Pro and Qwen3.8-27B participate in annotation construction or filtering, so rescoring with these two models provides limited independence from benchmark curation. Because the benchmark is a central contribution, stronger evidence is needed that its measured improvements correspond to human judgments of semantic and temporal correctness.

Questions for authors
  • Attribution of external gains: Can you report results on the external benchmarks using the identical planning, editing, and Ref2VA pipeline with untrained Qwen3.5-9B, SFT, GRPO, and ReKey reflection agents? Please keep generation settings and budgets matched and provide paired uncertainty estimates. This would establish whether learned reflection itself generalizes beyond ReKey-Bench.

  • Validation of benchmark scoring: Were method identifiers anonymized before evaluation despite their inclusion in the published template? Can you provide a blinded human audit of edited keyframes and generated videos, including evaluator–human agreement and whether the main method rankings are preserved?

3 comments

Response to W1 & Q1 (External Comparison Fairness):

Thank you for your comments. We compare reflection agents within the same generation pipeline and evaluate six baselines using Ref2VA. We have summarized these comparisons in Sec. 4.2 and provided the experimental details and results in Appendix C.5 (Tables 13 and 14) of the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Reflection Agent Comparison. We compare Base Qwen3.5-9B, SFT, GRPO, Label-guided OPSD, and ReKey within the same generation pipeline. All agents share the planning procedure, initial assets, image editor, video generator, generation settings, and reflection and editing budgets. We evaluate 250 T2V-CompBench cases and 300 VBench-2.0 cases, selecting 50 cases per dimension using a fixed-seed hash ordering of case IDs. ReKey achieves the highest mean score on both benchmarks, with 95% confidence intervals for paired differences reported in Tab. R1.

Tab. R1. Reflection agent comparison within the same ReKey pipeline. Scores follow the original benchmark aggregation over each method’s valid results and are reported as percentages. Confidence intervals are for paired differences computed as ReKey minus each comparator on cases with valid scores for both methods, expressed in percentage points.

Reflection agentT2V-CompBench ↑95% CIVBench-2.0 ↑95% CI
Base66.56[+0.95, +6.70]66.36[−0.01, +8.78]
SFT66.96[+0.44, +6.40]68.56[−2.01, +6.76]
GRPO67.09[+0.60, +6.19]66.21[+0.37, +8.41]
Label-guided OPSD66.52[+1.08, +6.61]67.00[−0.67, +7.33]
ReKey69.93—70.33—

Ref2VA Baseline Comparison. We compare ReKey with six baselines under the same two-stage Ref2VA generation procedure, using one reference image for coarse video generation and three keyframes for final video generation. The baselines share the same unedited candidate keyframes, while ReKey verifies and corrects its candidates. Each optimization method retains its own procedure and obtains any required video feedback from Ref2VA-generated videos. As shown in Tab. R2, ReKey achieves the highest average score on both benchmarks when the baselines also use Ref2VA.

Tab. R2. Comparison with six baselines using Ref2VA on all cases in the selected benchmark dimensions. ReKey (w/o reflection) omits keyframe verification and rectification. Scores and confidence intervals follow the convention in Tab. R1.

MethodT2V-CompBench ↑95% CIVBench-2.0 ↑95% CI
ReKey (w/o reflection)65.94[+0.38, +2.80]58.35[+2.41, +8.69]
SCMAPR65.10[+1.06, +3.83]59.07[+1.37, +8.30]
SSPO64.71[+1.58, +4.09]59.53[+0.79, +7.94]
PhyT2V65.82[+0.48, +2.96]57.77[+2.96, +9.30]
NEWTON64.58[+1.66, +4.27]60.00[+0.56, +7.15]
VPO65.57[+0.60, +3.31]61.17[−1.05, +6.47]
ReKey67.54—63.89—

Response to W2 & Q2 (Evaluator Robustness and Human Agreement):

Thank you for your comments. We clarify evaluator anonymization and assess the scores using two additional evaluators and a blinded human study. We have summarized these evaluations in Sec. 4.1 and provided the expanded evaluator comparison and human evaluation in Appendix C.3 (Tables 9, 10, and 11) of the updated submission, with changes highlighted in blue\color{blue}{\text{blue}}.

Evaluator Robustness. We anonymized method identifiers in the inputs to all automated evaluators, including those used for the original benchmark results. We rescore the existing outputs of all seven reflection agents with GLM-5.3-Flash and DeepSeek V4.1 Flash, covering 400 video cases and 784 independent keyframe tasks. As shown in Tab. R3, ReKey achieves the highest keyframe rectification and video improvement scores under all four evaluators.

Tab. R3. Scores across evaluators. Each cell reports SrectI\mathcal{S}_{\mathrm{rect}}^{I} / SimpV\mathcal{S}_{\mathrm{imp}}^{V}.

Reflection agentGemini 3.1 ProQwen3.8-27BGLM-5.3-FlashDeepSeek V4.1 Flash
Base0.281 / 0.3680.301 / 0.5610.341 / 0.5850.298 / 0.568
SFT0.351 / 0.3770.415 / 0.5640.478 / 0.5550.356 / 0.547
GRPO0.379 / 0.3860.420 / 0.5860.498 / 0.5920.322 / 0.559
Label-guided OPSD0.274 / 0.3670.313 / 0.5740.377 / 0.5680.238 / 0.549
QA-guided OPSD0.356 / 0.3590.411 / 0.5660.433 / 0.5610.381 / 0.563
QA+-guided OPSD0.367 / 0.3780.373 / 0.5750.439 / 0.5790.327 / 0.484
ReKey0.440 / 0.4380.489 / 0.6160.552 / 0.6000.393 / 0.611

Human Evaluation. We conduct a blinded study on a category-balanced sample of 50 ReKey-Bench cases across five methods. The materials are divided into eight batches, each independently rated by three annotators (24 total). Factual yes/no answers are aggregated by majority vote. We compare automated and human method rankings using Spearman (SROCC) and Kendall’s τb\tau_b (KROCC). Tabs. R4 and R5 compare our primary evaluator, Gemini 3.1 Pro, with human judgments. ReKey ranks first under both, with stronger ranking agreement for keyframes than for videos. Fleiss’ κ\kappa [1], computed from raw votes including the original-keyframe and Direct-video conditions, is 0.847 for keyframes and 0.796 for videos.

Tab. R4. Human and Gemini keyframe evaluation. All rows report SrectI\mathcal{S}_{\mathrm{rect}}^{I}.

EvaluatorBaseSFTGRPOLabel-guidedReKeySROCCKROCC
Gemini0.2700.2450.3820.2080.4700.9000.800
Human0.2270.2320.2420.1360.435——

Tab. R5. Human and Gemini video evaluation. All rows report SimpV\mathcal{S}_{\mathrm{imp}}^{V}.

EvaluatorBaseSFTGRPOLabel-guidedReKeySROCCKROCC
Gemini0.3050.2140.3620.2820.4300.5000.400
Human0.6250.6010.5950.5870.698——

[1] Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619

Thank you again for your constructive comments and suggestions.

Thank you once again for your positive assessment of our work. Your suggestions have been very helpful. As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed your concerns. If any questions remain, please let us know. We would be happy to provide further clarification and continue the discussion.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely appreciate the reviewers' time and effort in evaluating our paper. Reviewers 3eTq and UMNc both rated ReKey 3 (weak accept), highlighting the value of ReKey-Bench and the evidence for QAE-guided training. However, the remaining reviewer FCCe maintained a score of 2 (weak reject) without responding to our detailed rebuttal. Below, we summarize Reviewer FCCe's main concerns along with our corresponding responses.

Concern 1 (Comparison Fairness). Do ReKey’s gains persist under the same generation pipeline?

  • Response. We compared ReKey with six baselines under a shared Ref2VA procedure. We also compared five reflection agents on the same 550 external cases. ReKey ranked first on both benchmarks in both comparisons. Sec. 4.2 summarizes the results, and Appendix C.5 (Tables 13 and 14) reports paired confidence intervals.

Concern 2 (Backend Generalization). Does ReKey retain its advantage with different image editors and video generators?

  • Response. ReKey ranked first in both keyframe rectification and video improvement with the Qwen-Image2.1 editor and in video improvement with both LTX-2.3 and Wan2.1-VACE (Sec. 4.1 and Appendix C.6, Tables 16 and 17). Two additional automated evaluators ranked ReKey first on both metrics when rescoring the original benchmark outputs, as did human raters in a blinded study of 50 cases (Appendix C.3, Tables 9–11).

Concern 3 (Methodological Novelty). How does ReKey differ from prior reflection and distillation methods?

  • Response. R³-Refiner learns to rectify individual images, while Vision-OPD and Video-OPSD use privileged visual evidence for question answering. ReKey uses atomic question–answer–evidence (QAE) guidance to train an agent to specify what to correct and what to preserve across successive keyframes (Sec. 3 and Appendix A, Table 5). With distillation held fixed, QAE outperformed length-matched QA+ in verification, rectification, and downstream video improvement (Sec. 4.3, Table 4).

We believe our additional experiments and clarifications have addressed the three concerns raised by Reviewer FCCe. Thank you for your time and consideration.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.