- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper frames NR-IQA as a "scalable skill" of frozen VLMs: instead of fine-tuning per dataset, Q-Harness elicits per-attribute subscores from a frozen VLM via fixed prompts and maps them to MOS with a ridge calibration layer fitted on 100–200 labels. Experiments cover five IQA benchmarks, cross-dataset transfer, label-skew robustness, and backbone scaling.
Critical strengths
- The few-shot deployment scenario is real and well motivated; "calibrate rather than retrain" is a clean idea.
- The controlled ablation is well designed: it shows the gain is semantic rather than variance reduction from repeated queries.
Critical weaknesses
-
"Few-shot" is never pinned down, and the required budget likely differs by problem. The paper claims 100–200 labels suffice, but how small is small enough? Fig. 7 shows diminishing returns around n ≈ 200–300 in-domain, yet this is measured only on the same task family. Different target domains/problems (e.g., AIGC vs. synthetic distortion vs. in-the-wild) plausibly require different label budgets — there is no per-problem budget analysis or validation experiment. The headline "100–200 labels" is thus an average over heterogeneous settings without characterizing when it fails.
-
Attributes are fixed and hand-picked, never defined — in tension with the "scalable skill" claim. The 20 features come from two hand-designed prompts whose attributes are merely "selected from the quality vocabulary of existing IQA databases". There is no formal definition of what an attribute is, no selection criterion, and no mechanism to extend the schema.
-
The two prompt variants overlap, but the paper doesn't explain why. The 13-attribute and 5-attribute variants share attributes (sharpness, noise, lighting), producing correlated features, yet I find no justification anywhere in the paper — no explanation of why overlap is used, and no ablation of "two overlapping prompts" vs. "one comprehensive prompt." This is a basic design choice affecting the 20-dim feature space and needs both an explanation and an ablation.
-
The comparison is too thin, and the core idea may not be novel in substance. The few-shot comparison includes essentially one baseline (fine-tuned Q-Align). Two omissions are critical: (a) Classical same-idea baseline. The exact same recipe — per-attribute indicators + a lightweight fitter (e.g., SVM/ridge) on a few labels — can be built from traditional NR-IQA attribute features (or CLIP-IQA-style antonym probes, which the related work cites but never compares). If classical attribute indicators + SVM achieve comparable or better results, the contribution reduces to "use a VLM as the feature extractor," which is not a fundamental solution to few-shot / unseen-domain quality assessment. (b) The problem is not fundamentally solved. The paper claims to address quality evaluation under scarce labels and unseen domains, but with a fixed schema, a single-domain calibration layer, and SRCC-only transfer evidence, this remains an incremental engineering combination rather than a root-level innovation.
-
the calibration layer adds only +0.055 mean SRCC over the uncalibrated overall rating, and the fusion gain shrinks monotonically with backbone scale (Table 2, +0.131 at 0.8B → +0.028 at 397B) , what does this mean...
Questions for authors
- What is the per-problem label budget requirement? Can you show budget curves for each of the five domains separately?
- Why do the two prompt variants overlap? Please provide the missing justification and a one-prompt ablation.
- How does a classical baseline (per-attribute NR indicators + SVM/ridge, same label budget) compare?
- With the same backbone (Qwen3.5-9B), does fine-tuning still lose to the harness?
- How are attributes defined and selected — and how would the system handle a quality dimension outside the fixed 20?
2 comments
We are wondering if the reviewers have any other concerns
Dear Reviewer HKFh,
Thank you again for your constructive comments. As the author-reviewer discussion period ends in about one day, we would like to kindly ask whether our responses have addressed your concerns.
Following your suggestions, we added per-dataset label-budget analyses, clarified attribute selection and extension, and provided prompt ablations and comparisons with classical features and same-backbone fine-tuning.
We would be happy to clarify any remaining questions or concerns.
Best regards, The Authors
Response to Reviewer HKFh
Thank you again for your detailed and constructive review. We hope our response and additional experiments have addressed your questions about the label budgets, attribute design, and comparisons. If any concerns remain, we would be happy to provide further clarification.