- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ReKey-Bench, a benchmark for evaluating keyframe verification, keyframe rectification, and downstream video generation in reflective video generation. The benchmark consists of 400 video cases and 784 keyframe samples. In addition, the paper proposes ReKey, which trains a keyframe reflection agent via on-policy self-distillation. A frozen teacher is provided with question-answer-evidence (QAE) as privileged context, while the student learns to perform keyframe verification and rectification using only the original inputs. Experiments based on the Qwen3.5 series show that ReKey outperforms multiple training and inference baselines in keyframe verification, rectification, and downstream video generation. The paper further evaluates ReKey on T2V-CompBench and VBench-2.0, and conducts ablation studies with different forms of teacher guidance to validate the effectiveness of QAE supervision.
Critical strengths
None
Critical weaknesses
The paper proposes ReKey-Bench and ReKey. ReKey-Bench consists of 400 video cases and 784 keyframe samples, and evaluates reflective video generation at three stages: keyframe verification, keyframe rectification, and downstream video improvement. ReKey, meanwhile, employs on-policy self-distillation with QAE as privileged context, enabling the model to perform more reliable keyframe reflection and correction at inference time without requiring additional evidence inputs. Experiments show that ReKey outperforms multiple baselines in keyframe verification, rectification, and downstream video generation, with further support from QAE ablations, bootstrap confidence intervals, and evaluations using different evaluators. However, the methodological novelty is relatively limited, as it mainly builds on existing OPSD and privileged-evidence distillation paradigms. In addition, on external T2V benchmarks, ReKey uses keyframe-conditioned generation while some comparison methods rely on different generation interfaces, making it difficult to attribute the observed gains solely to ReKey itself. Furthermore, the benchmark depends on a fixed image editor, video generator, and VLM evaluator, and its generalization across different backbones remains to be further validated.
Questions for authors
-
Fairness of the external T2V comparison. ReKey uses keyframe-conditioned video generation based on Ref2VA, while some baselines use FL2VA. Could the authors provide comparisons under the same generation pipeline to more accurately isolate the contribution of ReKey itself?
-
Generalization across backends. ReKey-Bench relies on fixed image editors, video generators, and evaluation models. Could the authors clarify whether ReKey maintains its relative advantage when using different image editors or video generators?
-
Novelty relative to existing reflection and OPSD methods. Given that R³-Bench has already studied reflection/rectification in visual generation, and recent works such as Vision-OPD and Video-OPSD also employ on-policy self-distillation with privileged visual evidence, what is the core methodological novelty of ReKey beyond applying these ideas to sequential keyframe-based video generation?
6 comments
Response to Q2 (Backend Generalization):
Thank you for your comments. We evaluate an alternative image editor, two video generators, and two additional evaluators. We have summarized these results in Section 4.1 and provided the editor and generator comparisons in Appendix C.6 (Tables 16 and 17) and the evaluator comparison in Appendix C.3 (Table 9) of the updated submission, with changes highlighted in .
Alternative Image Editor. We evaluate Qwen-Image2.1 with prompt enhancement (PE) on 784 independent keyframe tasks and 400 sequential video cases per method. All reflection agents share the same editing configuration. As shown in Tab. R3, ReKey achieves the highest keyframe rectification and video improvement scores with this configuration. The editor and generator comparisons use Qwen3.5-9B reflection agents and Qwen3.8-27B for evaluation.
Tab. R3. Results with Qwen-Image2.1 + PE on ReKey-Bench.
| Reflection agent | ↑ | ↑ |
|---|---|---|
| Base | 0.3168 | 0.5198 |
| SFT | 0.3649 | 0.5631 |
| GRPO | 0.3459 | 0.5549 |
| Label-guided OPSD | 0.3019 | 0.5266 |
| ReKey | 0.4193 | 0.5863 |
Alternative Video Generator. We evaluate LTX-2.3 on all 400 ReKey-Bench cases and Wan2.1-VACE on a fixed 100-case subset. For each generator, video improvement is measured relative to its own text-only Direct baseline on the same cases. ReKey achieves the highest video improvement score with both generators (Tab. R4).
Tab. R4. Video improvement scores with alternative generators. U3 uses the original three keyframes without reflection or editing. Reflection agents use Qwen3.5-9B.
| Method | LTX-2.3 ↑ | Wan2.1-VACE ↑ |
|---|---|---|
| Direct (text-only) | 0.0000 | 0.0000 |
| U3 (original keyframes) | 0.5549 | 0.2948 |
| SFT | 0.5933 | 0.3361 |
| GRPO | 0.6075 | 0.3401 |
| Label-guided OPSD | 0.5604 | 0.2975 |
| ReKey | 0.6386 | 0.3603 |
Additional Evaluator. We rescore the original benchmark outputs with GLM-5.3-Flash and DeepSeek V4.1 Flash, covering 784 independent keyframe tasks and 400 video cases. ReKey retains the highest scores on both metrics under both evaluators (Tab. R5).
Tab. R5. Scores under additional evaluators. Each cell reports / .
| Reflection agent | GLM-5.3-Flash | DeepSeek V4.1 Flash |
|---|---|---|
| Base | 0.341 / 0.585 | 0.298 / 0.568 |
| SFT | 0.478 / 0.555 | 0.356 / 0.547 |
| GRPO | 0.498 / 0.592 | 0.322 / 0.559 |
| Label-guided OPSD | 0.377 / 0.568 | 0.238 / 0.549 |
| ReKey | 0.552 / 0.600 | 0.393 / 0.611 |
Response to Q3 (Distinction from Prior Work):
Thank you for your comments. We clarify the role of atomic QAE guidance and its distinction from prior reflection and on-policy distillation methods. We have clarified QAE's role in Section 3, expanded the analysis of teacher guidance under the same OPSD procedure in Section 4.3, and added a comparison with prior work in Appendix A (Table 5) of the updated submission, with changes highlighted in .
Our contributions. We introduce a diagnostic benchmark and an evidence-grounded approach to keyframe reflection. We study whether an agent can identify semantic and temporal errors, correct them while preserving satisfied requirements, and improve the resulting video. ReKey-Bench connects these questions through keyframe verification, rectification, and downstream video evaluation. The original-keyframe baseline, stage-level results, and analysis of corrected versus newly violated requirements help distinguish successful reflection from unnecessary or ineffective editing.
Distinction from prior work. R³-Refiner [1] learns visual verification and rectification through GRPO with verification and edited-image rewards. Vision-OPD [2] and Video-OPSD [3] use privileged crops and annotated evidence frames, respectively, to improve visual question answering. ReKey builds on on-policy distillation with atomic QAE guidance for corrective responses. Each QAE entry connects a visual requirement, its satisfaction status, and supporting image evidence. Together, these entries specify what needs correction and what should be preserved, including planned changes relative to the preceding keyframe. The teacher uses this guidance to supervise the student's verdict, explanation, and editing instruction, without requiring QAE at inference. The selected keyframe then provides the visual context for the next stage. Tab. R6 summarizes these distinctions and the corresponding evaluation components.
Tab. R6. Comparison with prior work. Locations refer to this paper.
| Feature | R³-Bench [1] | Vision-OPD [2] | Video-OPSD [3] | ReKey |
|---|---|---|---|---|
| Verification and rectification | ✓ | — | — | ✓ (Sec. 2.1) |
| Privileged on-policy distillation | — | ✓ | ✓ | ✓ (Sec. 3, Fig. 4) |
| Plan-conditioned keyframe reflection | — | — | — | ✓ (Sec. 2.1) |
| Atomic QAE teacher guidance | — | — | — | ✓ (Sec. 3, Fig. 4) |
| Keyframe-to-video evaluation | — | — | — | ✓ (Sec. 2.3, Fig. 2, Tab. 1) |
OPSD Alone Does Not Explain the Gains. As shown in Table 4 of the original main paper, reproduced below as Tab. R7, we hold the backbone, training data, budget, and OPSD procedure fixed while varying only the teacher guidance. OPSD with verification labels, atomic QA, or length-matched QA+ performs below QAE on all four metrics. Compared with QA+, QAE improves F1 from 0.628 to 0.733, keyframe rectification from 0.367 to 0.440, and downstream video improvement from 0.378 to 0.438. These comparisons show that adopting OPSD with label or QA guidance does not reproduce the gains from QAE.
Tab. R7. Effects of teacher guidance on ReKey-Bench, reproduced from Table 4 of the original main paper. All variants use the same OPSD procedure. QA+ matches QAE's teacher-prompt length by repeating atomic QA.
| Teacher guidance | Acc. ↑ | F1 ↑ | ↑ | ↑ |
|---|---|---|---|---|
| Verification label | 0.719 | 0.545 | 0.274 | 0.367 |
| Atomic QA | 0.795 | 0.649 | 0.356 | 0.359 |
| QA+ (length-matched) | 0.781 | 0.628 | 0.367 | 0.378 |
| Atomic QAE (ReKey) | 0.832 | 0.733 | 0.440 | 0.438 |
[1] Junjie Wang et al. Benchmarking and Evolving Reason-Reflect-Rectify for Reflective Visual Generation. arXiv:2605.19639, 2026.
[2] Qianhao Yuan et al. Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation. arXiv:2605.18740, 2026.
[3] Ziyue Wang et al. Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models. arXiv:2608.27065, 2026.
Thank you for this suggestion. We have clarified these points in Sections 3 and 4.3 and Appendix A of the updated submission.
Thank you for your comments. We clarify the original backend selection and provide Ref2VA baseline and reflection-agent comparisons. We have added a summary of these comparisons to Sec. 4.2, with the backend clarification, experimental details, and results provided in Appendix C.5 (Tables 13 and 14) of the updated submission. The changes are highlighted in blue.
Original Setup. In the original experiments, configurations with reference keyframes use Ref2VA, while configurations without reference images use FL2VA in text-only
t2vamode. This follows the documented H3 interfaces, where Ref2VA requires a reference input and FL2VA supports text-only generation.Nevertheless, we agree that comparisons under a shared generation pipeline provide a more direct assessment of ReKey’s contribution. We therefore provide two complementary comparisons.
Ref2VA Baseline Comparison. We compare ReKey with six baselines under the same two-stage Ref2VA generation procedure, using one reference image for coarse video generation and three keyframes for final video generation. The baselines share the same unedited candidate keyframes, while ReKey verifies and corrects its candidates. Each optimization method retains its own procedure and obtains any required video feedback from Ref2VA-generated videos. As shown in Tab. R1, ReKey achieves the highest average score on both benchmarks when the baselines also use Ref2VA.
Tab. R1. Comparison with six baselines using Ref2VA on all cases in the selected benchmark dimensions. ReKey (w/o reflection) omits keyframe verification and rectification. Scores follow the original benchmark aggregation over each method’s valid results and are reported as percentages. Confidence intervals are for paired differences computed as ReKey minus each comparator on cases with valid scores for both methods, expressed in percentage points.
Reflection Agent Comparison. We compare Base Qwen3.5-9B, SFT, GRPO, Label-guided OPSD, and ReKey within the same generation pipeline. All agents share the planning procedure, initial assets, image editor, video generator, generation settings, and reflection and editing budgets. We evaluate a fixed subset of 250 T2V-CompBench cases and 300 VBench-2.0 cases, with 50 cases per dimension. Scores follow the original benchmark aggregation over each method’s valid results. We estimate paired 95% confidence intervals for ReKey’s improvement over each comparator using cases with valid scores for both methods.
Tab. R2. Reflection agent results on the fixed 550-case subset. Scores and confidence intervals follow the convention in Tab. R1.