- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies label-efficient image quality assessment and argues that pretrained MLLMs already have reasonably strong quality perception, while their main difficulty is aligning predictions with dataset-specific MOS scales. Based on this observation, the paper proposes LEAF, a two-stage framework. In the first stage, an MLLM teacher provides point-wise quality judgments and pair-wise preferences to train a lightweight student without human MOS labels. In the second stage, a small MOS-labeled subset is used to calibrate the student predictions. Experiments on two AIGC and two UGC datasets show that the method can achieve competitive IQA performance with substantially fewer MOS annotations.
Critical strengths
The main idea of separating quality perception from MOS calibration is simple but useful. The paper gives a reasonable motivation that MLLMs can preserve image quality ranking even when their absolute scores are not well aligned with dataset-specific MOS. The experiments cover both AIGC and UGC datasets, and the method performs consistently well in the low-label setting. In particular, the label-free results are strong compared with the reported label-free baselines. The paper also evaluates different teacher models and student backbones, which makes the experimental results more convincing. Another practical advantage is deployment efficiency. The MLLM teacher is only used offline, while inference is performed with the much smaller student model.
Critical weaknesses
My main concern is that the central claim that MLLMs are strong at perception but mainly weak at MOS calibration is demonstrated most directly on AGIQA-3K. It is not clear whether the same behavior holds equally well across other datasets, distortion types, and teacher models. The comparison in the weakly supervised setting is also somewhat difficult to interpret. LEAF uses 10% or 30% MOS, while several competing methods use substantially more labeled data. This supports the label-efficiency argument, but it does not provide a controlled comparison under the same annotation budget. The contribution of pair-wise supervision appears relatively small compared with point-wise distillation, while it requires additional teacher queries. A clearer analysis of the accuracy-cost trade-off would help justify this part of the framework. Finally, the calibration is still dataset-specific. A new dataset or quality distribution appears to require another MOS subset for Stage 2. Cross-dataset calibration or transfer experiments would make the perception-calibration argument stronger.
Questions for authors
- The main motivation is that MLLMs already have strong quality perception but are poorly calibrated to MOS. Figure 2 mainly demonstrates this on AGIQA-3K. Does the same pattern hold on AIGIQA-20K, KonIQ-10k, and SPAQ?
- How sensitive is LEAF to the choice of teacher? Table 3 compares several MLLMs on AGIQA-3K, but does a weaker teacher consistently lead to weaker results on the other datasets as well?
- The weakly supervised baselines often use around 70% MOS, while LEAF uses 10% or 30%. Could you provide comparisons where competing methods are trained with the same 10% or 30% MOS budget?
- The gain from pair-wise supervision in Table 2 is relatively small compared with point-wise supervision. Considering that pair-wise supervision requires additional teacher inference, what is the practical accuracy-cost trade-off of this component?
- The current method performs calibration separately for each dataset. What happens if the student calibrated on one dataset is directly evaluated on another dataset? A cross-dataset experiment would help show whether Stage 1 learns transferable quality perception.
- The results with 10% and 30% MOS are averaged over five random MOS subsets. Could you also report the standard deviation or confidence interval? Some of the differences between methods are fairly small.
- How does the performance of the final student compare directly with the MLLM teacher on all four datasets? This comparison would help clarify how much quality information is lost during distillation and how much is recovered by MOS calibration.
- In the label-free setting, the student does not use human MOS but still receives dense teacher-generated quality labels. Would “MOS-free” or “human-label-free” be a more precise description than “label-free”?