- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReVRA, a video restoration agent that dynamically selects restoration tools through iterative planning, execution, and evaluation. It introduces an evolving memory for tool selection and a DPO-trained VLM for pairwise quality assessment. Experiments across seven benchmarks report improvements over existing restoration methods and agents.
Critical strengths
-
The iterative plan–execute–evaluate framework is reasonable for handling compound degradations. Table 4 provides evidence that re-planning can improve restoration quality compared with fixed pipelines.
-
The evaluation covers seven benchmarks, including real-world UGC videos. I appreciate the aligned-tool comparisons in Table 3 and the efficiency analysis of evolving memory in Table 6.
Critical weaknesses
-
I noticed that the TaoUGC results for ReVRA in Table 1 exactly match the memory-free configuration in Table 6, rather than the proposed full IEM configuration. This raises questions about which system was actually evaluated. Also, the reported human preferences for brute-force and ReVRA (53%, 50%, and 3% ties) do not sum to 100%. I think these inconsistencies need clarification.
-
Appendix A.6 averages CLIPIQA, MUSIQ, DOVER, and negative NIQE without normalization. Since MUSIQ has a much larger numerical range, this baseline is effectively dominated by MUSIQ. Moreover, Table 5 compares the DPO-trained evaluator against untuned VLMs, making it difficult to distinguish the benefit of pairwise comparison from preference fine-tuning. Interestingly, the untuned VLMs perform substantially worse than the metric-based baseline on KVQ.
Questions for authors
-
Why do the TaoUGC results in Table 1 match the memory-free configuration in Table 6 rather than the full ReVRA configuration? Which results correspond to the final system?
-
Why are the IQA/VQA metrics averaged without normalization in Appendix A.6? Could the authors clarify whether normalized or calibrated metric-based comparisons were evaluated?
-
Could the authors provide an ablation separating pairwise comparison from DPO fine-tuning? I think this is important for validating the claimed advantage of the comparison-based evaluator.
4 comments
Response to Reviewer 3BEJ (Part 2/3)
W2&Q2: Evaluation of Normalized QA-based Methods
Appendix A.6 averages CLIPIQA, MUSIQ, DOVER, and negative NIQE without normalization. Since MUSIQ has a much larger numerical range, this baseline is effectively dominated by MUSIQ. Moreover, Table 5 compares the DPO-trained evaluator against untuned VLMs, making it difficult to distinguish the benefit of pairwise comparison from preference fine-tuning. Interestingly, the untuned VLMs perform substantially worse than the metric-based baseline on KVQ.
Why are the IQA/VQA metrics averaged without normalization in Appendix A.6? Could the authors clarify whether normalized or calibrated metric-based comparisons were evaluated?
Response: Even through the QA-based's accuracy may be improved by e.g., normalization, our DPO-trained evaluator remains more accurate on both datasets, because aggregating differentmetrics still may not cover the variance of the degradations.
The original QA-based method followed the calculation used by existing agents[1] to maintain consistency with their evaluation protocol. We agree that directly averaging metrics with different numerical ranges can give MUSIQ disproportionate influence. We therefore normalize the four metrics, reverse the direction of NIQE, and then average them with equal weights:
, where and denote the mean and standard deviation of each metric over the entire test set. The comparison is shown below:
| Dataset | Qwen3-VL | Gemini3.1-pro | QA-based(Raw) | QA-based(Normalized) | Ours |
|---|---|---|---|---|---|
| KVQ | 62.06% | 64.25% | 92.51% | 96.82% | 97.84% |
| VideoLQ | 63.91% | 62.96% | 64.40% | 61.71% | 74.36% |
Normalization improves QA-based accuracy on KVQ from 92.51% to 96.82%, reducing its gap to our evaluator to 1.02 percentage points. On VideoLQ, accuracy decreases from 64.40% to 61.71%, while our evaluator achieves 74.36%, yielding a gap of 12.65 percentage points. These no-reference metrics measure different aspects of visual quality using different standards, and their scores can vary widely across video content and degradation types. Normalization puts their scores on similar scales, but does not make them reflect human preferences in the same way. A fixed combination of these metrics may therefore fail to capture how different degradations affect perceived quality across videos. Our pairwise comparison directly judges which video looks better, avoiding the need to aggregate these different metrics, while DPO aligns the judgments with human preferences.
Revision: We have replaced the original QA-based results in Table 5 with the normalized results and clarified the calculation and normalization procedure in Appendix A.6.
Response to Reviewer 3BEJ (Part 3/3)
W2&Q3: Ablation Separating Pairwise Comparison from DPO Fine-tuning
Appendix A.6 averages CLIPIQA, MUSIQ, DOVER, and negative NIQE without normalization. Since MUSIQ has a much larger numerical range, this baseline is effectively dominated by MUSIQ. Moreover, Table 5 compares the DPO-trained evaluator against untuned VLMs, making it difficult to distinguish the benefit of pairwise comparison from preference fine-tuning. Interestingly, the untuned VLMs perform substantially worse than the metric-based baseline on KVQ.
Could the authors provide an ablation separating pairwise comparison from DPO fine-tuning? I think this is important for validating the claimed advantage of the comparison-based evaluator.
Response: Preference fine-tuning improves both evaluation strategies, yet our trained pairwise comparison model remains more accurate when both use the same preference data. This supports the combination of pairwise comparison and preference learning beyond the benefit of additional training data alone.
To distinguish these contributions, we add absolute scoring as a control using the same Qwen3-VL backbone and human-annotated video pairs. Specifically, we evaluate both pairwise comparison and absolute scoring before preference fine-tuning, then train their counterparts using DPO and RL2R[2], respectively. The results are shown below:
| Assessment Strategy | Additional Training | KVQ | VideoLQ |
|---|---|---|---|
| Absolute scoring | 63.20% | 55.42% | |
| Relative comparison | 62.06% | 63.91% | |
| Absolute scoring | 80.96% | 62.97% | |
| Relative comparison | 97.84% | 74.36% |
Before preference fine-tuning, pairwise comparison achieves slightly higher accuracy overall: the two strategies perform similarly on KVQ, while pairwise comparison leads by 8.49 percentage points on VideoLQ. As the reviewer notes, the untuned VLM underperforms QA-based on KVQ, highlighting the importance of preference fine-tuning. Crucially, when both strategies use the same preference training data, our pairwise comparison model still outperforms absolute scoring by 16.88 and 11.39 percentage points on KVQ and VideoLQ, respectively. These results clarify both contributions: preference fine-tuning substantially strengthens the VLM’s quality assessment capability, and both additional preference training and relative comparison contribute to improving visual quality comparison accuracy.
Revision: We have added this ablation and its analysis to Table 9 in Appendix A.6 of the revised manuscript.
[1] Yingjie Zhou. . Q-agent: Quality-driven chain-of-thought image restoration agent through robust multimodal large language model. Arxiv.
[2] Tianhe Wu, et al. . Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank, Neurips.
Further Discussion with Reviewer 3BEJ
Dear Reviewer 3BEJ,
Thank you for your time and for your thoughtful and constructive comments. We have considered your suggestions carefully and respond to them below.
- We revise the TaoUGC results in Table 1 and clarify that they correspond to the final system with the IEM enabled.
- We clarify how the human preference statistics are computed and revise the relevant statements in Sec. 4.5 to avoid ambiguity.
- We provide an evaluation against normalized QA-based metrics, replace the original QA-based results in Table 5, and clarify the computation in Appendix A.6. We further explain why normalization alone does not align different metrics with human preferences.
- We extend the ablation separating the benefits of pairwise comparison from those of DPO fine-tuning, and add it to Appendix A.6.
As the discussion phase draws to a close, we hope our responses have addressed your concerns. Please let us know if any further questions remain. We sincerely appreciate your time and feedback.
Best regards,
The Authors.
Response to Reviewer 3BEJ (Part 1/3)
We thank the reviewer for the careful and detailed review, as well as for the positive recognition of our iterative plan–execute–evaluate framework and the breadth of our evaluation. Below we address the reviewer's concerns point by point: confirmation of the TaoUGC results in Table 1, the computation of the human preference statistics, an ablation separating the benefits of relative comparison from those of DPO training in the evaluation VLM, and a comparison of selection accuracy against normalized QA-based methods.
W1&Q1: Verification of TaoUGC Results in Table 1
Response: Table 1 evaluates the final system with IEM enabled. The TaoUGC row mistakenly copied the memory-free configuration results from Table 6, and we have corrected it to report the corresponding final-system results.
We thank the reviewer for identifying this error. The final system in Table 1 enables IEM, initializes it empty before evaluating each dataset, and updates it online during inference. However, the original TaoUGC row contained a transcription error, which we have corrected. To make clear which results correspond to which configuration, we list them below:
The corrected Table 1 results use an empty IEM initialization to prevent experience from carrying over across datasets. The full IEM configuration in Table 6 is instead pre-initialized on KVQ, GoPro, and VideoLQ before continuing online updates on TaoUGC. Both configurations use the same system components, but their initialization differs. This explains the difference between the corrected Table 1 results and the full IEM configuration in Table 6.
The corrected comparison on TaoUGC is shown below. CLIPIQA, MUSIQ, and DOVER remain higher than those of all compared methods; NIQE becomes slightly worse, but its ranking is unchanged. The correction therefore preserves the conclusions.
Revision: We have corrected the TaoUGC row in Table 1 and clarified the initialization settings in the captions of Tables 1 and 6. We have also checked all other reported results and found no further transcription errors.
W1: Human Preference Statistics
Response: The reported percentages overlap because the same 3% ties were counted toward both methods. Reporting mutually exclusive outcomes resolves this ambiguity: 50% prefer brute-force, 47% prefer ReVRA, and 3% are ties.
In the user study, 3% of cases were judged to have comparable perceptual quality. These cases were included in both preference percentages: the reported 53% for brute-force comprised 50% exclusive preferences plus 3% ties, while the reported 50% for ReVRA comprised 47% exclusive preferences plus the same 3% ties. We have adopted a clearer explain to avoid ambiguity.
Revision: We have revised the statement in Section 4.5 to make the counting explicit: “Brute-force and ReVRA were the exclusive preference in 50% and 47% of cases, respectively, with 3% ties.”