acceptodds
Under review as a conference paper at ICLR 2027

Q-Harness: Few-Shot No-Reference Image Quality Assessment as a Scalable Skill

Abstract

No-reference image quality assessment (NR-IQA) is often needed in settings where a new camera pipeline, restoration method, compression codec, or generative model creates a new target domain faster than a large mean-opinion-score (MOS) database can be collected. In such deployments, NR-IQA should be few-shot by design: a practical assessor should adapt from only a few labeled examples and be ready to use. Existing per-dataset fine-tuning does not match this need, because it spends scarce labels re-estimating a large perceptual model, overfits target-domain statistics, and must be repeated for every new domain. We instead frame NR-IQA as a scalable skill of foundation models: modern vision-language models (VLMs) already contain strong quality priors, and the right interface is to elicit and calibrate this skill rather than retrain it. We present Q-Harness, a harness system that turns a frozen general-purpose VLM into a few-shot quality assessor without backbone training. Given a small set of expert labels, it elicits interpretable per-attribute quality subscores (sharpness, noise, exposure, composition, etc.) and maps them to a final MOS through a lightweight ridge calibration layer fitted on those labels. A controlled ablation shows that the harness's gain is semantic: querying the same construct in multiple ways yields little benefit, irrelevant attributes receive near-zero weight, and performance improves as quality attributes become less redundant.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image quality assessment, low-level computer vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HKFhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper frames NR-IQA as a "scalable skill" of frozen VLMs: instead of fine-tuning per dataset, Q-Harness elicits per-attribute subscores from a frozen VLM via fixed prompts and maps them to MOS with a ridge calibration layer fitted on 100–200 labels. Experiments cover five IQA benchmarks, cross-dataset transfer, label-skew robustness, and backbone scaling.

Critical strengths
  • The few-shot deployment scenario is real and well motivated; "calibrate rather than retrain" is a clean idea.
  • The controlled ablation is well designed: it shows the gain is semantic rather than variance reduction from repeated queries.
Critical weaknesses
  • "Few-shot" is never pinned down, and the required budget likely differs by problem. The paper claims 100–200 labels suffice, but how small is small enough? Fig. 7 shows diminishing returns around n ≈ 200–300 in-domain, yet this is measured only on the same task family. Different target domains/problems (e.g., AIGC vs. synthetic distortion vs. in-the-wild) plausibly require different label budgets — there is no per-problem budget analysis or validation experiment. The headline "100–200 labels" is thus an average over heterogeneous settings without characterizing when it fails.

  • Attributes are fixed and hand-picked, never defined — in tension with the "scalable skill" claim. The 20 features come from two hand-designed prompts whose attributes are merely "selected from the quality vocabulary of existing IQA databases". There is no formal definition of what an attribute is, no selection criterion, and no mechanism to extend the schema.

  • The two prompt variants overlap, but the paper doesn't explain why. The 13-attribute and 5-attribute variants share attributes (sharpness, noise, lighting), producing correlated features, yet I find no justification anywhere in the paper — no explanation of why overlap is used, and no ablation of "two overlapping prompts" vs. "one comprehensive prompt." This is a basic design choice affecting the 20-dim feature space and needs both an explanation and an ablation.

  • The comparison is too thin, and the core idea may not be novel in substance. The few-shot comparison includes essentially one baseline (fine-tuned Q-Align). Two omissions are critical: (a) Classical same-idea baseline. The exact same recipe — per-attribute indicators + a lightweight fitter (e.g., SVM/ridge) on a few labels — can be built from traditional NR-IQA attribute features (or CLIP-IQA-style antonym probes, which the related work cites but never compares). If classical attribute indicators + SVM achieve comparable or better results, the contribution reduces to "use a VLM as the feature extractor," which is not a fundamental solution to few-shot / unseen-domain quality assessment. (b) The problem is not fundamentally solved. The paper claims to address quality evaluation under scarce labels and unseen domains, but with a fixed schema, a single-domain calibration layer, and SRCC-only transfer evidence, this remains an incremental engineering combination rather than a root-level innovation.

  • the calibration layer adds only +0.055 mean SRCC over the uncalibrated overall rating, and the fusion gain shrinks monotonically with backbone scale (Table 2, +0.131 at 0.8B → +0.028 at 397B) , what does this mean...

Questions for authors
  1. What is the per-problem label budget requirement? Can you show budget curves for each of the five domains separately?
  2. Why do the two prompt variants overlap? Please provide the missing justification and a one-prompt ablation.
  3. How does a classical baseline (per-attribute NR indicators + SVM/ridge, same label budget) compare?
  4. With the same backbone (Qwen3.5-9B), does fine-tuning still lose to the harness?
  5. How are attributes defined and selected — and how would the system handle a quality dimension outside the fixed 20?
2 comments
AuthorsOpenReview ↗

Response to Reviewer HKFh

Thank you again for your detailed and constructive review. We hope our response and additional experiments have addressed your questions about the label budgets, attribute design, and comparisons. If any concerns remain, we would be happy to provide further clarification.

AuthorsOpenReview ↗

We are wondering if the reviewers have any other concerns

Dear Reviewer HKFh,

Thank you again for your constructive comments. As the author-reviewer discussion period ends in about one day, we would like to kindly ask whether our responses have addressed your concerns.

Following your suggestions, we added per-dataset label-budget analyses, clarified attribute selection and extension, and provided prompt ablations and comparisons with classical features and same-backbone fine-tuning.

We would be happy to clarify any remaining questions or concerns.

Best regards, The Authors

Reviewer_kc2mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Q-harness, an approach for Image Quality Assessment that uses a frozen VLM with a fixed prompt to predict a low dimensional set of interprettable quality attributes (e.g., noise, sharpness, exposure, ...) with a simple regularized linear regression calibration layer to adapt to new domains. The idea is that adaptation to a new domain (or distortion) can be done with a few (e.g., hundreds) samples instead of large scale data collection. Unlike other approaches that may fine tune a model for a new domain, the authors demonstrate that for the same sample budget that their approach adapts & generalizes better.

The idea of predicting some component (interprettable) quality attributes / features isn't necessarily novel, nor is the use of ridge regression to align small set of features for IQA task.

Critical strengths

Experiments demonstrate the robustness of the method to skew in the labels (e.g., test various ways to sample small set of data to train the calibration layer) -- whereas fine-tuned approaches are very sensitive.

Critical weaknesses

The novelty is somewhat limited.

Experimental validation is largely based on one VLM that already performs pretty well on IQA in zero-shot mode. In Table 2, the gap between the zero-shot and "fusion" model is much smaller with their technique on InternVL3.5 and LLama-3.2 Vision.

Questions for authors

What amount of dataset specific training is done for the results in “Table 3”? You say "Unlike the matched-label experiment in Section 4.3, this comparison includes methods with different training regimes" and it is in a section on "COMPARISON WITH FULL-DATA". Do the compared methods listed in the table (train / test) using the same protocol?

The value of having two overlapping attribute sets was not evaluted. Why not just include the union of all attributes in one prompt?

Reviewer_zvDAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Q-Harness proposes a few-shot no-reference image quality assessment (NR-IQA) framework that uses a frozen vision-language model (VLM) to extract multiple quality-attribute scores and a lightweight ridge regression model to calibrate these scores to mean opinion scores (MOS). The paper evaluates the approach on five IQA datasets under limited-label settings, investigates cross-dataset transfer, and compares it with conventional NR-IQA methods and different VLM backbones. The reported results suggest that attribute-based calibration can improve quality prediction without updating the backbone model.

Critical strengths
  1. The paper addresses few-shot IQA under limited MOS annotations and explores an alternative to fine-tuning large models for each target dataset.
  2. The combination of a frozen VLM, attribute-level prompting, and ridge regression is straightforward to implement and does not require backbone optimization.
  3. The paper includes multiple IQA datasets, cross-dataset transfer experiments, attribute ablations, label-budget analysis, and backbone scaling studies.
Critical weaknesses
  1. The experimental evaluation does not sufficiently cover recent few-shot VLM-based IQA methods, particularly GRMP-IQA (ICCV 2025). Recent studies on lightweight adaptation of pretrained vision-language backbones are also highly relevant. The authors should clarify the scope of their comparison and include direct comparisons under matched annotation budgets and evaluation protocols where feasible.
  2. Several results in Table 3 differ substantially from some reported evaluations even on few-shot settings. The manuscript is suggested to provide enough detail about checkpoints, training data, target-domain supervision, and evaluation protocols to explain these discrepancies.
  3. Q-Harness combines a frozen VLM, prompt-based quality-attribute elicitation, and ridge regression. Although this design is simple and potentially useful, the manuscript does not yet sufficiently establish the benefit of its attribute-based representation over simpler alternatives. Comparisons using the same VLM backbone should include direct overall-score calibration, regression on frozen visual features, and parameter-efficient adaptation. The authors should also provide stronger evidence that the elicited attribute scores contain meaningful complementary information.
Questions for authors

I have several concerns regarding the manuscript that require clarification. First, the related-work section and experimental comparisons do not appear to sufficiently cover recent few-shot IQA methods, such as GRMP-IQA (ICCV 2025) or any other related few-shot IQA. Second, several baseline results differ substantially from those reported in the corresponding, and further details are needed to establish the fairness and reproducibility of these comparisons. Finally, the main methodological contribution of Q-Harness remains unclear beyond combining a frozen VLM, prompt-based quality-attribute elicitation, and ridge regression. Could the authors clarify these issues and provide stronger evidence that the proposed attribute-based calibration offers distinct advantages over existing methods and simpler alternatives?

2 comments
AuthorsOpenReview ↗

Response to Reviewer zvDA

Thank you again for your constructive review. We hope our response and additional experiments have addressed your concerns about the comparisons, evaluation protocols, and attribute-based representation. If any questions remain, we would be happy to provide further clarification.

AuthorsOpenReview ↗

We are wondering if the reviewers have any other concerns

Dear Reviewer zvDA,

Thank you again for your constructive comments. As the author-reviewer discussion period ends in about one day, we would like to kindly ask whether our responses have addressed your concerns.

Following your suggestions, we compared Q-Harness with GRMP-IQA under a matched label budget, clarified the baseline protocols, and added same-backbone comparisons and analyses of the attribute scores.

We would be happy to clarify any remaining questions or concerns.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.