- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies image aesthetic quality assessment in a joint setting where one multimodal LLM both generates a natural-language quality description and predicts a numerical score. The authors argue that prior work treats description generation and scoring as decoupled tasks, breaking the inherent consistency between them (the description should be the reasoning basis for the score, and the score a quantitative summary of the description), while naive single-sequence joint training suffers from optimization conflicts due to heterogeneous target formats and imbalanced supervision. They propose SAGA, built on Qwen2.5-VL-7B-Instruct, which embeds two task-specific expert branches whose "aesthetic experience latent spaces" are shaped by a scene–attribute bipartite graph distilled by a plug-and-play Scene-Attribute Recognition Module.
Critical strengths
- A well-motivated problem with direct supporting evidence. The description/scoring coupling is an important and under-explored issue in IQA/IAA. Crucially, the paper does not leave the motivation as rhetoric: the description-length sensitivity study (Sec. 4.3, Fig. 5) shows that vanilla joint SFT collapses (PLCC 0.140 on AIGCIQA2023 with 300-word descriptions) while SAGA remains stable across description lengths. This is concrete evidence that the claimed optimization conflict exists and that the proposed formulation mitigates it — the strongest part of the paper.
- A strong and unusually complete empirical package. SAGA achieves SOTA scoring on three datasets spanning aesthetic assessment (PARA) and technical IQA (SPAQ, AIGCIQA2023), best overall description similarity, and additionally provides balanced-multitask analysis (App. B), a human preference study, per-attribute breakdowns, and a 5.9–8.9% latency reduction versus separately trained models. The efficiency result makes the joint model practically attractive rather than only a benchmark winner.
Critical weaknesses
- The description ground truth is entirely LLM-generated and is never validated against human judgments, which undermines half of the paper's claims. All reference descriptions are produced by Gemini conditioned on the expert score, and the description metric (BGE cosine + character n-gram against these references) measures similarity to that same distribution. Two problems follow: (i) we do not know whether the references themselves are faithful, diverse, or expert-level. The 5-user study compares model outputs against each other, not the references against human-written descriptions; (ii) the metric rewards reproducing the pseudo-label distribution, so "description SOTA" may partly mean "closest to Gemini's phrasing". A human-written reference subset plus a metric–human correlation analysis would materially strengthen the paper.
- The gains are not cleanly attributed to the proposed coupling. (a) Scoring is formulated as autoregressive token generation, but the paper never compares this parameterization against a regression head or discrete-level scoring (e.g., Q-Align-style) under matched settings, and it does not report how PLCC/SRCC are computed from the generated text or the parse success rate. (b) The SFT collapse in the length study (PLCC 0.140) looks more like a loss-balancing/training artifact than a fundamental property of decoupled training — no loss-weighted, gradient-surgery, or curriculum baseline is tried, so the strawman risk is real. (c) The scene–attribute "graph" is a complete bipartite graph from which single (scene, attribute) subgraphs are extracted, on which the 4-layer GCN largely reduces to a learned embedding lookup; no analysis of learned edge weights or of what message passing adds beyond per-(scene, attribute) embeddings is provided.
Questions for authors
How exactly are PLCC/SRCC computed from the generated score text (decimal tokenization, parsing), and what fraction of generations parse successfully? Please also report a regression-head variant under identical settings. Can you validate the Gemini-generated reference descriptions against human-written descriptions on a subset, and report the correlation of your description metric with human judgment? In the description-length study, did you try loss weighting, gradient surgery (e.g., PCGrad), or curriculum strategies for the SFT baseline? Would vanilla SFT still collapse? What does the GCN add over per-(scene, attribute) learned embeddings? Are edge weights learned? Any analysis (e.g., ablating message passing vs. node lookup) would help.
3 comments
(4) Clarification on Edge Weights and Task Scope
In this paper, the edge weights are fixed to uniform weights and are not learned. Since the scene–attribute pair is already determined for each sample during inference, forward propagation is restricted to relevant edges to avoid interference from irrelevant attributes. This does not mean nodes are isolated: the shared global embedding and GCN parameters are updated across different pairs, implicitly fusing cross-pair information. In future work, for multi-attribute tasks within a single sequence, learnable edge weights will be necessary, for which we will consider attention networks like GAT/GATv2.
(5) Clarification on Scoring Extraction
We constrain the scoring format using a fixed output template: the model first outputs "The score is", followed by a floating-point score (e.g., "The score is 4.5."). Score extraction is performed at the text level rather than relying on token-level alignment. We use regular expressions to match the numerical value following "The score is" in the decoded generated text, convert it to a floating-point number, and then compute the PLCC and SRCC against the ground-truth labels. Across all test samples, no samples were discarded due to parsing failures.
(6) Clarification on Scoring Parameterization: Dual-NTP vs. Regression Head
We have added a regression-head variant under identical settings: the description branch remains NTP, and only the scoring branch is replaced with an MSE regression head. The results show that, under this coupling architecture, NTP + regression scoring performs comparably to the baseline overall, but underperforms our original dual-NTP paradigm. This indicates that, within our coupling architecture, joint training with the dual-NTP paradigm is better suited to this task than a regression head, which justifies our design choice. A possible explanation is that since both the description and scoring branches adopt NTP, they are more consistent in terms of loss formulation and representation space. This makes the optimization objectives more compatible during joint training and leads to a more natural alignment in the coupling representations. Finally, we have included these results in the revised manuscript and limited our conclusion to the comparison at the scoring parameterization level.
| Method | Dataset | PLCC | SRCC | Text |
|---|---|---|---|---|
| Regression Head | PARA | 0.942 | 0.943 | 1.518 |
| Regression Head | SPAQ | 0.855 | 0.859 | 1.645 |
| Regression Head | AIGCIQA2023 | 0.757 | 0.772 | 1.589 |
| SAGA (Dual-NTP) | PARA | 0.955 | 0.954 | 1.537 |
| SAGA (Dual-NTP) | SPAQ | 0.912 | 0.906 | 1.679 |
| SAGA (Dual-NTP) | AIGCIQA2023 | 0.810 | 0.811 | 1.625 |
Dear Reviewer MEQb,
Thank you for your valuable time in reviewing our manuscript. We have addressed each of the concerns point by point in our response. Please let us know if anything remains unclear—we would be happy to provide further clarification.
We sincerely thank the reviewer for the thorough review and constructive feedback. Below, we respond to each comment in turn, with clarifications, additional experiments, and revisions where applicable.
(1) Human Inspection of Gemini-Generated Descriptions and Validation of the Similarity Metric
We conducted a human inspection of the descriptions generated by Gemini. Specifically, we combined MLLMs and human evaluators to rate the descriptions on a 1–5 scale (1 = poor, 2 = fair, 3 = moderate, 4 = good, 5 = excellent) across four dimensions: factual accuracy, hallucination, consistency, and professionalism. Both the MLLMs and the human evaluators were given the same scoring rubric and asked to rate according to it. The MLLMs included GPT-5.4, Qwen3.8-Max, and DeepSeek-V4-Pro, and the human evaluation involved five volunteers. Samples were uniformly drawn from different score ranges, totaling 200 samples. The results show that both the MLLMs and the human evaluators assigned scores above 4 to the Gemini-generated descriptions, with an average score of 4.55. In addition, for the inference results on 339 samples, we report the overall similarity, semantic similarity, n-gram similarity, and human ratings, and compute the SRCC between each metric and human judgments. The results show that all these metrics are significantly correlated with human judgments: combined similarity achieves an SRCC of 0.7872 (p = 9.9e-73), semantic similarity 0.6996 (p = 4.0e-51), and n-gram similarity 0.7587 (p = 1.1e-64). However, these metrics cannot serve as the sole evaluation criterion. We therefore further include a user study in the paper.
(2) Weighted SFT Partially Mitigates but Does Not Resolve the Collapse
We agree that reporting the collapse of vanilla joint SFT alone is insufficient to rule out the possibility that “simple loss balancing” could resolve the issue. We therefore added a score-token-weighted weighted-SFT baseline. This baseline was trained under the same tuning budget and evaluation protocol as SAGA, and we tested two weight ratios (1:2 and 1:4). The results show that weighted SFT mitigates the collapse at 300 words, but it does not fully resolve the stability issue: performance still drops considerably at 300 words. Moreover, the weight ratio cannot be selected optimally a priori, and the optimal weight drifts across datasets and description lengths, limiting reliability. In contrast, SAGA remains superior in both cross-length stability and overall performance, without introducing length-dependent weighting hyperparameters. In the revised manuscript, we have limited our claim to “an empirical phenomenon of vanilla joint SFT in this setting” and “weighted SFT can partially mitigate the issue but is insufficient and hyperparameter-sensitive.”
(3) Clarification on GCN vs. Node-Lookup / Per-pair Embeddings
We clarify that our GNN does not learn an independent embedding for each (scene, attribute) pair. Instead, all scene and attribute nodes share a global learnable embedding matrix and shared GCN parameters. Nodes are not isolated; they are updated by samples from different pairs during training, thereby implicitly aggregating cross-scene and cross-attribute information. Compared with node-lookup baselines, the GCN adds: (1) local cross-node message passing and shared multi-hop transformations; (2) a structural prior for each sample; and (3) implicit cross-pair information fusion during training.
In the ablation in Table 5, we remove the GCN and directly use an embedding-only node lookup. The full model significantly outperforms the embedding-only model on all datasets. Furthermore, in our response to reviewer hBjD Q4, we added parameter-matched baselines without graph convolution (embedding+Linear and embedding+Attention) to demonstrate that the gains are not merely due to additional parameters.