- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents PACE, a training-free framework for open-ended image quality assessment. Its central premise is that direct multimodal scoring can conflate overall visual quality with performance on a specific evaluation dimension. PACE addresses this problem by constructing explicit evaluation protocols through concept decomposition, visual question generation, and iterative critique. It combines protocol-based scoring with relative comparison against human-rated reference images. The framework is evaluated on conventional IQA, structural integrity, context-dependent aesthetics, and six newly constructed perceptual dimensions. The results show substantial improvements on several open-ended dimensions.
Critical strengths
-
Adapting image evaluation to new, user-specified dimensions is an important problem. Formulating adaptation as explicit protocol construction provides a plausible way to make evaluation criteria more specific and inspectable.
-
Promising empirical improvements over the same backbone. On the six open-ended dimensions, PACE improves average PLCC/SRCC from 0.493/0.501 to 0.714/0.666 over Qwen2.5-VL. The gains on text rendering, tactile satisfaction, and human vitality are encouraging.
-
The paper includes multiple evaluation settings, component ablations, an additional backbone, and runtime measurements. The six-annotator study and reported inter-annotator agreement provide useful support for the human ratings.
Critical weaknesses
-
Scope of adaptation and dependence on human calibration. The formulation conditions evaluation on a user-specified dimension and reuses a protocol initially grounded in one image. The claimed low human overhead depends on substantial reuse of calibrated dimensions. If evaluation requests frequently introduce genuinely new dimensions, human calibration would be required for nearly every request, even though annotators rate reference images rather than the test images themselves.
-
The evidence for broad open-ended generalization remains limited. The central benchmark contains only 30 images per dimension, covers six author-selected dimensions, and uses a single image generator followed by manual filtering. It only provides limited evidence for robustness across arbitrary dimensions, image sources, and content distributions.
-
The ablations do not establish the mechanism behind the reported reduction in holistic bias. Table 5 compares Track 1, Track 2, and their combination on conventional IQA benchmarks using PLCC/SRCC. However, the central holistic-bias claim concerns the open-ended Tier-4 setting and the HOR metric.
Questions for authors
Please see the weaknesses. I would consider raising my rating if the authors address the concerns.
7 comments
Thank you for the additional experiments. The evaluation and ablations address part of my concerns. Two questions remain:
- Although a quality dimension can be shared across images, what ensures that a protocol grounded in one image and its calibration anchors remain valid across diverse contexts? When is recalibration required?
- The ablations report PLCC/SRCC rather than HOR. Could you report component-wise HOR with explicit calibration settings to distinguish improvements from protocol-based evaluation versus reference comparison or score rescaling? These clarifications would strengthen the interpretation of the results.
Thank you for your timely reply and for raising these helpful questions. We hope the following results address your concerns.
Q1 — Protocol and anchor reuse
As described in Sec. 4.3, the initial image provides visual grounding, while the generated protocol captures dimension-level criteria rather than image-specific details, allowing reuse across images under the same dimension. For efficiency, PACE reuses existing protocols and anchors by default.
Anchors can be recalibrated when the quality range changes, while protocols are reconstructed when new visual contexts introduce criteria not covered by the original probes. This balances efficiency and adaptability without retraining.
Q2 — Component-wise HOR and calibration settings
We further computed HOR for the Tier-4 component ablations (GT ≤ 3), including an additional score-rescaling control:
| Configuration | HOR ↓ |
|---|---|
| Direct Qwen2.5-VL | 38.3% |
| Direct + Anchor Rescaling | 34.6% |
| Track 1 Only | 11.1% |
| Track 2 Only | 16.0% |
| Without Critic | 9.9% |
| Full PACE | 8.6% |
Calibration settings: Direct + Anchor Rescaling maps direct predictions to the human-rated anchor range without protocol evaluation or visual reference comparison. Track 1 Only scores directly on the original 1–5 scale, without anchor comparison or rescaling. Track 2 Only uses anchor-based comparison and score calibration, while Full PACE combines both tracks (Eq. 10).
Score rescaling alone reduces HOR only from 38.3% to 34.6%, whereas Track 1 reduces it to 11.1% without calibration. This confirms that protocol evaluation, rather than score rescaling alone, drives most of the HOR reduction. Reference comparison provides additional benefits, with Full PACE achieving the lowest HOR (8.6%).
Thank you for recognizing the promising gains on open-ended dimensions. We hope our response will resolve your concerns.
W1 — Human calibration cost. In practical IQA, a quality dimension is typically applied to multiple images, rather than introducing an entirely new dimension for every evaluation request. PACE therefore requires only four candidate images to be rated once per new dimension; the resulting two anchors and evaluation protocol are then reused (Sec. 4.4). Even when new dimensions arise, this is substantially less costly than collecting a large MOS dataset and fine-tuning a dedicated scorer for each task.
Moreover, Track 1 alone provides much of the Tier-4 improvement without relative anchor comparison (PLCC/SRCC: 0.634/0.597 vs. direct scoring 0.493/0.501; see W3), while the anchors further align scores with human preferences.
W2 (Generalization beyond Tier-4). To evaluate generalization beyond our original six dimensions, we further tested PACE on 12 additional dimensions from EvalMuse-40K, which includes diverse prompts and images generated by multiple T2I models. The results compare direct scoring with PACE:
Across all 12 dimensions, macro PLCC improves from 0.249 to 0.338, and macro SRCC from 0.208 to 0.311. Both correlations improve on all 12 dimensions, spanning diverse visual attributes and fine-grained generation requirements. These results further support PACE's generalization beyond the original Tier-4 evaluation set.
W3 (Tier-4 ablations and holistic bias). We added component ablations directly on the six open-ended Tier-4 dimensions, using the same evaluation setting:
The results show that protocol-based evaluation is a major contributor: Track 1 alone substantially improves over direct scoring, while Critic refinement and dual-track fusion further improve the final correlations. Complementing these ablations, the original Q-Align vs. PACE evaluation shows that HOR decreases from 44.44% to 8.64% and BalancedHOR from 23.36% to 7.73% (see brPK W1), supporting the framework's effectiveness in reducing over-scoring on open-ended dimensions.