acceptodds
Under review as a conference paper at ICLR 2027

SAGA: Single-Sequence Coupling of Description and Scoring via Scene-Attribute Graphs based Aesthetic Learning

Abstract

Image quality assessment has evolved from a simple numerical regression to integrate score prediction with explanatory text generation. Existing methods usually treat these two objectives as completely independent tasks and train them in a decoupled manner, which may break the inherent causal relationship between the two tasks. Yet, directly modeling both tasks within a single generation sequence often leads to optimization conflicts and training instability due to the heterogeneity of target formats and imbalanced supervision signals. In this paper, we propose a novel Scene-Attribute Graphs based Aesthetic learning (SAGA) framework. Dual expert branches are fused within SAGA, based on their task-specific aesthetic experience latent spaces, to enhance the fusion model's capacity of capturing intricate aesthetic preferences. Thus, SAGA can effectively unify description generation and quality scoring within a single sequence. Extensive experiments show that SAGA significantly outperforms the state-of-the-art methods on both description generation and quality scoring, while exhibiting more balanced multitask performance. Furthermore, compared with models trained separately, SAGA maintains strong reasoning capability, while achieving notably lower inference latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, IQA, IAA, VLMs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MEQbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies image aesthetic quality assessment in a joint setting where one multimodal LLM both generates a natural-language quality description and predicts a numerical score. The authors argue that prior work treats description generation and scoring as decoupled tasks, breaking the inherent consistency between them (the description should be the reasoning basis for the score, and the score a quantitative summary of the description), while naive single-sequence joint training suffers from optimization conflicts due to heterogeneous target formats and imbalanced supervision. They propose SAGA, built on Qwen2.5-VL-7B-Instruct, which embeds two task-specific expert branches whose "aesthetic experience latent spaces" are shaped by a scene–attribute bipartite graph distilled by a plug-and-play Scene-Attribute Recognition Module.

Critical strengths
  1. A well-motivated problem with direct supporting evidence. The description/scoring coupling is an important and under-explored issue in IQA/IAA. Crucially, the paper does not leave the motivation as rhetoric: the description-length sensitivity study (Sec. 4.3, Fig. 5) shows that vanilla joint SFT collapses (PLCC 0.140 on AIGCIQA2023 with 300-word descriptions) while SAGA remains stable across description lengths. This is concrete evidence that the claimed optimization conflict exists and that the proposed formulation mitigates it — the strongest part of the paper.
  2. A strong and unusually complete empirical package. SAGA achieves SOTA scoring on three datasets spanning aesthetic assessment (PARA) and technical IQA (SPAQ, AIGCIQA2023), best overall description similarity, and additionally provides balanced-multitask analysis (App. B), a human preference study, per-attribute breakdowns, and a 5.9–8.9% latency reduction versus separately trained models. The efficiency result makes the joint model practically attractive rather than only a benchmark winner.
Critical weaknesses
  1. The description ground truth is entirely LLM-generated and is never validated against human judgments, which undermines half of the paper's claims. All reference descriptions are produced by Gemini conditioned on the expert score, and the description metric (BGE cosine + character n-gram against these references) measures similarity to that same distribution. Two problems follow: (i) we do not know whether the references themselves are faithful, diverse, or expert-level. The 5-user study compares model outputs against each other, not the references against human-written descriptions; (ii) the metric rewards reproducing the pseudo-label distribution, so "description SOTA" may partly mean "closest to Gemini's phrasing". A human-written reference subset plus a metric–human correlation analysis would materially strengthen the paper.
  2. The gains are not cleanly attributed to the proposed coupling. (a) Scoring is formulated as autoregressive token generation, but the paper never compares this parameterization against a regression head or discrete-level scoring (e.g., Q-Align-style) under matched settings, and it does not report how PLCC/SRCC are computed from the generated text or the parse success rate. (b) The SFT collapse in the length study (PLCC 0.140) looks more like a loss-balancing/training artifact than a fundamental property of decoupled training — no loss-weighted, gradient-surgery, or curriculum baseline is tried, so the strawman risk is real. (c) The scene–attribute "graph" is a complete bipartite graph from which single (scene, attribute) subgraphs are extracted, on which the 4-layer GCN largely reduces to a learned embedding lookup; no analysis of learned edge weights or of what message passing adds beyond per-(scene, attribute) embeddings is provided.
Questions for authors

How exactly are PLCC/SRCC computed from the generated score text (decimal tokenization, parsing), and what fraction of generations parse successfully? Please also report a regression-head variant under identical settings. Can you validate the Gemini-generated reference descriptions against human-written descriptions on a subset, and report the correlation of your description metric with human judgment? In the description-length study, did you try loss weighting, gradient surgery (e.g., PCGrad), or curriculum strategies for the SFT baseline? Would vanilla SFT still collapse? What does the GCN add over per-(scene, attribute) learned embeddings? Are edge weights learned? Any analysis (e.g., ablating message passing vs. node lookup) would help.

3 comments

We sincerely thank the reviewer for the thorough review and constructive feedback. Below, we respond to each comment in turn, with clarifications, additional experiments, and revisions where applicable.

(1) Human Inspection of Gemini-Generated Descriptions and Validation of the Similarity Metric

We conducted a human inspection of the descriptions generated by Gemini. Specifically, we combined MLLMs and human evaluators to rate the descriptions on a 1–5 scale (1 = poor, 2 = fair, 3 = moderate, 4 = good, 5 = excellent) across four dimensions: factual accuracy, hallucination, consistency, and professionalism. Both the MLLMs and the human evaluators were given the same scoring rubric and asked to rate according to it. The MLLMs included GPT-5.4, Qwen3.8-Max, and DeepSeek-V4-Pro, and the human evaluation involved five volunteers. Samples were uniformly drawn from different score ranges, totaling 200 samples. The results show that both the MLLMs and the human evaluators assigned scores above 4 to the Gemini-generated descriptions, with an average score of 4.55. In addition, for the inference results on 339 samples, we report the overall similarity, semantic similarity, n-gram similarity, and human ratings, and compute the SRCC between each metric and human judgments. The results show that all these metrics are significantly correlated with human judgments: combined similarity achieves an SRCC of 0.7872 (p = 9.9e-73), semantic similarity 0.6996 (p = 4.0e-51), and n-gram similarity 0.7587 (p = 1.1e-64). However, these metrics cannot serve as the sole evaluation criterion. We therefore further include a user study in the paper.

EvaluatorFactual AccuracyHallucinationConsistencyProfessionalism
GPT-5.44.77004.67504.47504.3750
Qwen3.8-Max4.26004.07504.52004.6800
DeepSeek-V4-Pro4.59004.43504.62504.8050
Human4.51004.56004.62504.7550

(2) Weighted SFT Partially Mitigates but Does Not Resolve the Collapse

We agree that reporting the collapse of vanilla joint SFT alone is insufficient to rule out the possibility that “simple loss balancing” could resolve the issue. We therefore added a score-token-weighted weighted-SFT baseline. This baseline was trained under the same tuning budget and evaluation protocol as SAGA, and we tested two weight ratios (1:2 and 1:4). The results show that weighted SFT mitigates the collapse at 300 words, but it does not fully resolve the stability issue: performance still drops considerably at 300 words. Moreover, the weight ratio cannot be selected optimally a priori, and the optimal weight drifts across datasets and description lengths, limiting reliability. In contrast, SAGA remains superior in both cross-length stability and overall performance, without introducing length-dependent weighting hyperparameters. In the revised manuscript, we have limited our claim to “an empirical phenomenon of vanilla joint SFT in this setting” and “weighted SFT can partially mitigate the issue but is insufficient and hyperparameter-sensitive.”

methods50 words100 words200 words300 words
Vanilla joint SFT0.670 / 0.7210.759 / 0.8110.694 / 0.7980.140 / 0.300
Weighted SFT (1:2)0.731 / 0.7760.727 / 0.7470.733 / 0.7390.670 / 0.671
Weighted SFT (1:4)0.714 / 0.7420.748 / 0.7640.426 / 0.7590.704 / 0.712
SAGA0.765 / 0.7760.810 / 0.8110.754 / 0.7690.781 / 0.792

(3) Clarification on GCN vs. Node-Lookup / Per-pair Embeddings

We clarify that our GNN does not learn an independent embedding for each (scene, attribute) pair. Instead, all scene and attribute nodes share a global learnable embedding matrix and shared GCN parameters. Nodes are not isolated; they are updated by samples from different pairs during training, thereby implicitly aggregating cross-scene and cross-attribute information. Compared with node-lookup baselines, the GCN adds: (1) local cross-node message passing and shared multi-hop transformations; (2) a structural prior for each sample; and (3) implicit cross-pair information fusion during training.

In the ablation in Table 5, we remove the GCN and directly use an embedding-only node lookup. The full model significantly outperforms the embedding-only model on all datasets. Furthermore, in our response to reviewer hBjD Q4, we added parameter-matched baselines without graph convolution (embedding+Linear and embedding+Attention) to demonstrate that the gains are not merely due to additional parameters.

(4) Clarification on Edge Weights and Task Scope

In this paper, the edge weights are fixed to uniform weights and are not learned. Since the scene–attribute pair is already determined for each sample during inference, forward propagation is restricted to relevant edges to avoid interference from irrelevant attributes. This does not mean nodes are isolated: the shared global embedding and GCN parameters are updated across different pairs, implicitly fusing cross-pair information. In future work, for multi-attribute tasks within a single sequence, learnable edge weights will be necessary, for which we will consider attention networks like GAT/GATv2.

(5) Clarification on Scoring Extraction

We constrain the scoring format using a fixed output template: the model first outputs "The score is", followed by a floating-point score (e.g., "The score is 4.5."). Score extraction is performed at the text level rather than relying on token-level alignment. We use regular expressions to match the numerical value following "The score is" in the decoded generated text, convert it to a floating-point number, and then compute the PLCC and SRCC against the ground-truth labels. Across all test samples, no samples were discarded due to parsing failures.

(6) Clarification on Scoring Parameterization: Dual-NTP vs. Regression Head

We have added a regression-head variant under identical settings: the description branch remains NTP, and only the scoring branch is replaced with an MSE regression head. The results show that, under this coupling architecture, NTP + regression scoring performs comparably to the baseline overall, but underperforms our original dual-NTP paradigm. This indicates that, within our coupling architecture, joint training with the dual-NTP paradigm is better suited to this task than a regression head, which justifies our design choice. A possible explanation is that since both the description and scoring branches adopt NTP, they are more consistent in terms of loss formulation and representation space. This makes the optimization objectives more compatible during joint training and leads to a more natural alignment in the coupling representations. Finally, we have included these results in the revised manuscript and limited our conclusion to the comparison at the scoring parameterization level.

MethodDatasetPLCCSRCCText
Regression HeadPARA0.9420.9431.518
Regression HeadSPAQ0.8550.8591.645
Regression HeadAIGCIQA20230.7570.7721.589
SAGA (Dual-NTP)PARA0.9550.9541.537
SAGA (Dual-NTP)SPAQ0.9120.9061.679
SAGA (Dual-NTP)AIGCIQA20230.8100.8111.625

Dear Reviewer MEQb,

Thank you for your valuable time in reviewing our manuscript. We have addressed each of the concerns point by point in our response. Please let us know if anything remains unclear—we would be happy to provide further clarification.

Reviewer_hBjDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework that jointly performs image quality scoring and description generation within a single sequence. Built upon Qwen2.5-VL-7B, SAGA introduces two task-specific expert branches and scene-attribute graphs to learn aesthetic representations, while sharing the decoder for joint optimization. Experiments on PARA, SPAQ, and AIGCIQA2023 demonstrate competitive scoring accuracy.

Critical strengths

The experiments cover three datasets from both IQA and IAA domains, with extensive architecture ablations in Table 4. The human evaluation in Table 3 also provides additional evidence of the quality of the generated descriptions.

Critical weaknesses

Although improving consistency between descriptions and scores is a central motivation, the training objective in Section 3.3 simply combines two task-specific losses without an explicit consistency constraint. The experiments do not directly measure whether the generated descriptions agree with their corresponding scores, making it difficult to verify this central claim.

Questions for authors

1 Please clarify how the generated descriptions influence score prediction within the single sequence, since the scoring loss in Section 3.3 conditions only on preceding score tokens.

2 Table 5 shows that separate training achieves better scoring performance on SPAQ (PLCC 0.921 versus 0.912), so the performance trade-off introduced by joint training deserves further discussion.

3 Since SARM is trained using images from the evaluated datasets, please clarify whether the benchmark test images are excluded from its training process.

4 A parameter-matched baseline without graph convolution would help distinguish the benefits of scene-attribute graph modeling from those of additional model capacity.

5 Please clarify whether the inference times in Table 5 include the computational overhead of SARM and GNN processing.

2 comments

We thank the reviewer for the thorough review and valuable feedback. We appreciate both the recognition of our work and the concerns raised. Below, we respond to each point to clarify our contributions and their applicability and to address these concerns.

(Q1) The coupling between the two tasks is not an explicit constraint; it mainly occurs in the shared decoder. During training, the decoder carries the joint gradient flow from both tasks, so the two tasks influence each other through the shared parameters and become aligned at the representation level. We conducted a decoder-frozen variant in Table 5, where the two tasks are completely gradient-independent and each updates only its own expert module. The results show that performance drops for both tasks across all datasets, indicating that the gradients through the shared decoder carry bidirectional cross-task interaction. In addition, Appendix B and the user study report that, under the same data and training configuration, SAGA exhibits more consistent task performance than other methods.

(Q2) We acknowledge that separate training achieves a slightly higher score PLCC on SPAQ (0.921 vs. 0.912, a gap of about 0.009). This gap is dataset-specific; on the other metrics, SAGA outperforms separate training on every dataset. In addition, separate training paradigms often require multi-stage training, which in turn necessitates multi-step inference. In contrast, SAGA’s modeling is more holistic, yields smoother inference, and its output format is better suited to scenarios that require both descriptions and scores. We will add a discussion of this comparison in the revised manuscript.

(Q3) SARM is trained only on the training split of each dataset, with test-set images completely excluded. We will explicitly emphasize this point in the corresponding section. In addition, we also report in Appendix C the robustness of SARM under various perturbations.

(Q4) We agree that the “w/o Graph” variant in Table 5 also reduces the parameter count, making it hard to disentangle the contributions of the graph modeling from those of model capacity. To address this, we additionally include two parameter-matched, graph-convolution-free variants based on linear networks and attention networks. Both variants underperform the version with the scene–attribute graph. Since the parameter counts are matched, this degradation can only be attributed to removing the graph. Therefore, scene–attribute graph modeling is better suited to explainable IQA tasks that are scene-dependent and attribute-dependent.

ModelDatasetTextPLCCSRCC
LinearSPAQ1.5730.8490.844
LinearPARA1.3830.9210.920
LinearAIGCIQA20231.1720.6620.651
AttentionSPAQ1.6050.8910.882
AttentionPARA1.4040.9080.909
AttentionAIGCIQA20231.2520.7500.762

(Q5) The inference time in Table 6 includes the full computational overhead of both SARM and the GNN, and is measured as the end-to-end inference time from input image to final output. The reported times therefore reflect the true inference cost of the complete method.

Dear Reviewer hBjD,

Thank you for your valuable time in reviewing our manuscript. We have addressed each of the concerns point by point in our response. Please let us know if anything remains unclear—we would be happy to provide further clarification.

Reviewer_PvBjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly addresses the training imbalance between textual description generation and numerical scoring in image quality and aesthetic assessment. The proposed solution augments a VLM framework with ViT-based scene recognition information and two task-specific branches. The paper further supplements and evaluates data on three open-source datasets, and conducts experiments on description quality, score prediction, and human preference.

Critical strengths

The paper addresses an important practical gap by jointly producing interpretable quality descriptions and calibrated scores in a consistent output, which is valuable for real IQA applications where a score alone is insufficient. It also provides a reasonably broad empirical validation across three datasets, including scoring metrics, description metrics, human preference, latency comparison, and ablation studies, supporting the claim that SAGA is more balanced than naive joint fine-tuning.

Critical weaknesses

The data construction for the text description task lacks convincing validation, such as human verification or cross-validation with multiple models, and it does not sufficiently demonstrate the consistency between the generated descriptions and the predicted scores. Both training and evaluation rely on VLM-generated text, which may affect the paper’s core conclusions.

In addition, all experiments are conducted by training and validating on the same datasets; compared with other baseline methods, the paper lacks comparisons in terms of parameter scale and does not provide sufficient evidence for generalization.

Questions for authors

1.SARM’s ViT and the other branches are trained on PARA/SPAQ/AIGCIQA2023 and are also evaluated on these same datasets. The paper lacks out-of-domain comparisons with other baseline methods, making it difficult to demonstrate the generalization ability of the proposed method. Could you provide cross-dataset or out-of-domain evaluation results?

2.The paper lacks comparisons of ViT configurations and overall model parameter counts. If the proposed model introduces substantial extra parameters, the comparison against other baselines may be unfair. Please provide parameter-scale comparisons and efficiency analysis

3.Could the authors include more recent perceptual or aesthetic scoring models as additional baselines, such as ArtiMuse、UniPercept?

6 comments

We thank the reviewer for the meticulous review and valuable feedback. Below, we clarify and respond to each point in order.

(1) Clarification on OOD Generalization

To further evaluate the generalization capability of our method, we have added OOD evaluations on two multi-attribute scoring datasets: AADB and APDDv2. We compared the generalization performance of the base model, SFT, Q-Insight, and our SAGA. The results show that SAGA consistently outperforms the other comparison methods on both OOD datasets across PLCC and SRCC. We have included these OOD evaluation results in the revised manuscript.

ModelAADB (PLCC / SRCC)APDDv2 (PLCC / SRCC)
Base0.205 / 0.2650.066 / 0.058
SFT0.343 / 0.4590.255 / 0.264
Q-Insight0.254 / 0.2770.178 / 0.153
SAGA (Ours)0.433 / 0.4670.282 / 0.268

(2) Clarification on Parameter Counts

We clarify that SARM adopts the ViT-Base-Patch16 architecture, consisting of 12 Transformer encoder layers, a hidden dimension of 768, 12 attention heads, and an intermediate dimension of 3072, which contains approximately 86M parameters. In addition, each expert branch consists of four GCN layers (feature dimension 300) and six attention layers (dimension 3054). These components are designed to enhance graph-based feature representation and attention-based feature interaction. As detailed in Table 6, we present the inference efficiency of SAGA, reporting both the overall inference time across different datasets and the inference latency per single sample.

To further address the concern regarding parameter scale and fairness, we have added two additional baselines in the revision: LLaVA-1.5-13B and SAGA (Qwen2.5-VL-3B). The results show that merely scaling up parameters does not guarantee better performance: LLaVA (13B) yields a lower PLCC (0.800) compared to 7B models. Conversely, our smaller SAGA (base-3B) achieves a superior PLCC of 0.949, outperforming all other baselines. These newly added results clearly demonstrate that SAGA's performance gain stems from our architectural design and parameter efficiency, rather than an unfair advantage from a larger parameter scale.

ModelPLCCSRCCText
Qwen2.5-VL-7B0.9240.9421.536
LLaVA-1.5-7B0.9380.9391.536
LLaVA-1.5-13B0.8000.9441.538
SAGA (base-7B)0.9550.9541.537
SAGA (base-3B)0.9490.9451.516

(3) Inclusion of Additional Baselines: ArtiMuse and UniPercept

We have included two recent aesthetic scoring models, ArtiMuse and UniPercept, as additional baselines for comparison. Considering the differences in their modeling paradigms and task formulations, we adopt a prompt-based approach to elicit their scores for each specific attribute dimension. Furthermore, we have added discussions of these methods to the Related Work and Experimental sections.

ModelPARASPAQAIGCIQA2023
UniPercept0.104 / 0.7610.011 / 0.5050.036 / 0.420
ArtiMuse0.577 / 0.5790.117 / 0.3370.107 / 0.183

(4) Human Inspection of Gemini-Generated Descriptions

We conducted a human inspection of the descriptions generated by Gemini. Specifically, we combined MLLMs and human evaluators to rate the descriptions on a 1–5 scale (1 = poor, 2 = fair, 3 = moderate, 4 = good, 5 = excellent) across four dimensions: factual accuracy, hallucination, consistency, and professionalism. Both the MLLMs and the human evaluators were given the same scoring rubric and asked to rate according to it. The MLLMs included GPT-5.4, Qwen3.8-Max, and DeepSeek-V4-Pro, and the human evaluation involved five volunteers. Samples were uniformly drawn from different score ranges, totaling 200 samples. The results show that both the MLLMs and the human evaluators assigned scores above 4 to the Gemini-generated descriptions, with an average score of 4.55.

EvaluatorFactual AccuracyHallucinationConsistencyProfessionalism
GPT-5.44.77004.67504.47504.3750
Qwen3.8-Max4.26004.07504.52004.6800
DeepSeek-V4-Pro4.59004.43504.62504.8050
Human4.51004.56004.62504.7550

Dear Reviewer PvBj,

Thank you for your valuable time in reviewing our manuscript. We have addressed each of the concerns point by point in our response. Please let us know if anything remains unclear—we would be happy to provide further clarification.

Thank you for your reply. I have re-examined the updated manuscript and noticed that, on the SRCC metric, the results of SFT are only marginally distinguishable from SAGA. Could you please explain this issue? Specifically, does it imply that the improvement in the metric derives primarily from the data level rather than from the architectural modification?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.