acceptodds
Under review as a conference paper at ICLR 2027

Faithful Textual Explanations for Interpretable Image Aesthetic Assessment

Abstract

In image aesthetic assessment (IAA), explanations typically rely on evaluative terms like harmonious or nostalgic. However, these terms describe the overall impression rather than specific regions of an image, and they lack ground-truth annotations for evaluation. Existing concept bottleneck models (CBMs) cannot be easily applied here because they rely on localized, verifiable visual concepts. Furthermore, selecting concepts based solely on their correlation with aesthetic ratings often fills explanations with paraphrases of "good image" or with descriptions of basic image content rather than aesthetic quality. To address these limitations, we introduce FaTeX, an interpretable IAA model that generates faithful, grounded, and image-specific textual explanations without requiring manual concept annotations. FaTeX calculates aesthetic scores by summing contributions from named concept pairs generated by a large language model (LLM) and scored using CLIP. It controls for image content, filters out redundant paraphrases, and uses concepts that apply both across datasets and within specific content categories. A gating step is designed to give no weight to level-specific concepts whose subject does not appear in the image, so that they drop out of the explanation. Evaluated across six single- and multi-genre datasets, FaTeX trails the best non-interpretable models by .035 SRCC on average (at most .058) and outperforms existing attribute-based approaches. In human and LLM-based evaluations measuring explanation quality and image identification, FaTeX produces better and more specific explanations than existing concept-based methods, and is on par only where a baseline draws on expert style annotations. Code and materials are available at https://anonymous.4open.science/r/proj-FaTeX-E4FB.

Primary area:
interpretability and explainable AI
Keywords:
Image aesthetic assessment, Explainable AI, Interpretability, Concept bottleneck models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_w8CrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies interpretable image aesthetic assessment (IAA), where standard concept bottleneck assumptions are problematic because aesthetic concepts are evaluative, content-dependent, and lack ground-truth concept annotations. It proposes FaTeX, which constructs dataset-specific pools of antonymic textual concepts using an LLM, filters them by rating association after controlling for image content, and uses orthogonal matching pursuit to reduce redundancy. A frozen CLIP encoder supplies signed concept responses and an applicability gate, while a trainable image encoder allocates per-image concept weights and a magnitude. The prediction is an exact sum of named concept contributions. Experiments on six datasets show that FaTeX remains within 0.035 SRCC of the best predictive methods on average and improves over C2J, although PPA-T is more accurate on five of six datasets. Human and LLM studies on LAPIS and PARA generally prefer FaTeX's top-three concept summaries and find them more image-specific than PPA-T; C2J remains tied on LAPIS, where it uses expert style annotations. My main concern is that exact additivity establishes an arithmetic decomposition, but not necessarily semantic faithfulness: the learned weights and magnitude can carry unnamed task information, and the displayed top-three text covers only about one third of the total absolute contribution.

Critical strengths
  1. The problem formulation and research story are excellent. The paper clearly explains why aesthetic concepts lack the localization, objective labels, and content invariance assumed by standard CBMs. Content-controlled selection, a hierarchy of collection-wide and content-specific concepts, and per-image applicability gating are well motivated responses to these failures. This makes the work scientifically meaningful even though many individual building blocks have precedents.

  2. The empirical study is broad, transparent, and unusually thoughtful. It covers six datasets, two backbones, predictive and concept-based baselines, human and LLM evaluation, image-identification tests, and extensive ablations. Task 2 controls the distractors' content tag and rating band, making it a useful test of specificity. The authors also report unfavorable results, including that FaTeX is never the best predictive model, and provide detailed protocols and limitations that materially improve assessability and reproducibility.

Critical weaknesses
  1. Exact additivity is insufficient to establish semantic faithfulness. The learned allocation p(t|x) and magnitude rho(x) provide unrestricted image-dependent channels around the frozen concept response. The paper itself observes that borrowing another image's allocation lowers SRCC by roughly 0.05/0.10 and acknowledges that this may be unnamed task information. Without a dedicated concept/label leakage test or semantic interventions, large values of c_t show that a term is numerically important, not that the term's natural-language name captures the actual reason for the prediction. The deletion experiment inherits this limitation. Moreover, the evaluated top-three text covers only 31.5%/34.0% of absolute contribution and shows no values, so “the explanation is the prediction” is true only for the complete internal vector, not for the textual explanation presented to users.

  2. The human evidence is strong as a relative vocabulary comparison but limited as an end-to-end explanation evaluation. The study first restricts evaluation to test images on which every compared model predicts within half a test-set standard deviation of the target, yet does not report the eligible fraction. This can exclude difficult or failure cases. The explanation template uses the ground-truth low/average/high band rather than the model prediction, and only LAPIS and PARA are evaluated. Each comparison has two raters per image, uses forced choice without a neutral option, and the many pair-by-question intervals receive no multiplicity correction. The core ablations are also retrained only once. These choices do not invalidate the relative result, but they make the paper's general claims about generated explanations and statistical significance too broad.

Questions for authors
  1. Can the authors provide a dedicated information-leakage analysis that separates what can be predicted from d(x,t), p(t|x), and rho(x), and tests whether weights retain the target after concept responses or concept identities are randomized? Concept-level counterfactual interventions or a variant whose weights depend only on verified concept values would materially affect my view of the faithfulness claim.

  2. Since the displayed top-three concepts account for only 31.5%/34.0% of total absolute contribution and show no numerical values, what completeness or fidelity guarantee does the actual textual explanation provide? Please report its residual contribution or reconstruction error and compare different explanation lengths.

  3. What number and fraction of each test set pass the all-model error filter used for the human study? Do the main human results persist on an unfiltered random test sample when the template uses the model's predicted band rather than the ground-truth band? This would determine how far the reported explanation quality generalizes to deployment and model failures.

6 comments
AuthorsOpenReview ↗

Response to Reviewer w8Cr (Part 1/5)

We thank the reviewer for proposing, with deep expertise in interpretable AI, concrete ways to test our claims, such as leakage tests and counterfactual interventions. Thanks to these comments, the manuscript now states the faithfulness claim within a more accurate scope. We are also grateful that the reviewer recognized the integrity of this work, including our disclosure of unfavorable results. We respond to each point below. For Q1, additional experiments are still in progress; we answer the other points first and will post our response to Q1 as soon as they are complete.

  1. Since the displayed top-three concepts account for only 31.5%/34.0% of total absolute contribution and show no numerical values, what completeness or fidelity guarantee does the actual textual explanation provide? Please report its residual contribution or reconstruction error and compare different explanation lengths.

[Response] We agree. The exact decomposition holds for the full vector of concept contributions, and the three-concept text is a summary of it.

In short, the displayed three-concept text preserves the direction of the prediction (in 92% of images) and its ranking (SRCC .94 / .92 with the full prediction), but not its magnitude: about 60% of the deviation from the baseline is carried by concepts that are not named. k = 3 is a choice in favour of readability.

To quantify what the text guarantees, we measured, for each explanation length k, how well the sum SkS_k of the k largest contributions reproduces the prediction y^=a+∑tct\hat{y} = a + \sum_t c_t, where a is the baseline. We used the runs explained in the human study, over the full test sets (LAPIS 2,345, PARA 3,000), without retraining. Values are LAPIS / PARA.

Explanation length k13 (paper)510all
Share of $\Sigma$$\lvert c_t \rvert$ in the top k.15 / .15.32 / .34.42 / .46.60 / .641
Relative residual mean∣y^−a−Sk∣/mean∣y^−a∣\mathrm{mean}\lvert \hat{y}-a-S_k \rvert / \mathrm{mean}\lvert \hat{y}-a \rvert.80 / .82.60 / .60.48 / .47.31 / .290
Images where sign(Sk)=sign(y^−a)\mathrm{sign}(S_k) = \mathrm{sign}(\hat{y}-a)89% / 90%92% / 92%94% / 94%96% / 96%100%
SRCC(a+Sk,y^)\mathrm{SRCC}(a+S_k, \hat{y}).86 / .83.94 / .92.96 / .95.98 / .981

The residual falls gradually with k (still about 30% at k = 10), with no clear elbow. The 120 human-study images show the same pattern (relative residual at k = 3: .56 / .61; sign agreement 92% / 91%). The text therefore guarantees the direction and ranking of the prediction, not its magnitude. Where the magnitude is needed, the full contribution vector, for which the exact decomposition holds, can be shown with its values.

Restricting the text to a few concept names is to some extent unavoidable for an explanation that people read and judge. Label-Free CBM (Oikarinen et al., 2023) likewise presents only a subset of the contributing concepts, by name, in its human evaluation. Our k = 3 follows the same reasoning: the text is a readable summary of the full contribution vector.

[Action] We will add the table (k = 1, 3, 5, 10, all) to the appendix. We will revise the annotation "the explanation is the prediction" in Fig. 2 and the claim in the text to state that the exact decomposition holds for the full contribution vector, and that the displayed three-concept text is a summary that preserves the direction and ranking of the prediction but not most of its magnitude.

AuthorsOpenReview ↗

Response to Reviewer w8Cr (Part 2/5)

  1. What number and fraction of each test set pass the all-model error filter used for the human study? Do the main human results persist on an unfiltered random test sample when the template uses the model's predicted band rather than the ground-truth band? This would determine how far the reported explanation quality generalizes to deployment and model failures.

[Response] In short, 35.9% (LAPIS) and 52.6% (PARA) of test images passed the original filter. In a new human evaluation without the filter and with the predicted band, raters identified the image from FaTeX's explanations (Task 2) at least as often as in the original setting, and above chance even on images the model got wrong. Preference (Task 1) held on LAPIS, where FaTeX was chosen about as often on images the model got wrong as on the others, but narrowed on the PARA images the model got wrong, where the difference from PPA-T was no longer clear.

Images for the human study were drawn from those on which every compared model predicts within half a test-set SD of the target. The numbers and fractions that pass are:

DatasetTestPass (all models)Pass rate by band (low / average / high)
LAPIS2,345843 (35.9%)32.0% / 40.8% / 35.0%
PARA3,0001,577 (52.6%)43.2% / 69.4% / 45.4%

The filter keeps differences in predictive accuracy out of the comparison of explanations. As the reviewer notes, however, it limits our conclusions to images on which the models are roughly correct.

The original study cannot answer this question. All of its images are ones on which every model predicted correctly, so it contains no image on which the model is wrong. Also, the template's level word was the ground-truth band rather than the model's prediction. This kept the level word identical across the two explanations being compared, so that raters compared only the concepts; with predicted bands, the level word would change whenever the methods' predictions differ. We therefore ran an additional human study to measure the two things you ask about:

  • (a) Do the original conclusions hold as the explanations would be used (no filter, level word from the model's predicted band)?
  • (b) Are the explanations still useful on images the model gets wrong?

Tasks, questions, checks and exclusion criteria are those of the original study; only two things changed:

  • Images: without the filter, we drew 60 images at random from those FaTeX misses (|error| > 0.5 SD) and 60 from those it does not. A simple random sample of 120 images would contain missed images only at their share of the test set (16.3% on PARA, 35.7% on LAPIS), about 20 images on PARA. To examine model failures, which the question also raises, at the same time, we oversampled missed images so that (b) is answered with enough images. Weighting the two strata by their shares of the test set estimates the same quantities as an unfiltered random sample, and we answer (a) with this estimate (the "unfiltered (weighted estimate)" columns of the table).
  • Level word: each model's predicted band instead of the ground-truth band.

As this is a follow-up study, we reduced the raters per item from two to one, which widens the confidence intervals.

LAPIS: originalLAPIS: unfiltered (weighted estimate)LAPIS: model wrongPARA: originalPARA: unfiltered (weighted estimate)PARA: model wrong
Task 1: FaTeX preferred (Q0, vs w/o Tg∗T^{\ast}_g)63.7%63.5%66.7%65.4%70.3%55.0%
Task 1: same (vs PPA-T)71.2%77.9%80.0%65.0%56.4%46.7%
Task 1: same (vs C2J)56.2%58.9%60.0%69.6%72.5%68.3%
Task 2: FaTeX accuracy (chance 25%)37.5%45.5%43.3%36.2%37.5%41.7%

"Model wrong" is the stratum of 60 images on which FaTeX misses by more than 0.5 SD; "original" is the submitted human study (two raters per item).

(a) Unfiltered test images. In the weighted estimates, Task 2 accuracy is at or above the original setting. On LAPIS, the 95% CIs of the differences from all three methods exclude zero; on PARA, the differences from C2J and PPA-T have the same sign as before, with CIs that just include zero. In Task 1, LAPIS is close to the original setting. On PARA, FaTeX is preferred at or above the original rate over w/o Tg∗T^{\ast}_g and C2J, but its preference over PPA-T falls to 56.4% from 65.0%, with a CI that includes 50%.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer w8Cr (Part 3/5)

(continued from the previous comment)

(b) Images on which the model is wrong. In Task 2, on the 60 images the model misses, raters identified the correct image from FaTeX's explanation 43.3% of the time on LAPIS (95% CI 31.7–56.7%) and 41.7% on PARA (30.0–55.0%), above chance (25%) and no lower than on the 60 images it predicts well. In Task 1, LAPIS raters preferred FaTeX about as often on missed images as on the others, whereas on PARA FaTeX's advantage narrowed on missed images, to 46.7% against PPA-T (58.3% on well-predicted images).

With one rater per item, the confidence intervals are wider than in the original setting, and more comparisons have intervals that include chance or zero. Even so, FaTeX keeps the same direction of advantage as in the original setting in almost every comparison. The exception is Task 1 against PPA-T on PARA, where the preference reverses on images the model gets wrong (46.7%).

[Action] We will add the pass-rate table to the appendix. We will also add the unfiltered, predicted-band human evaluation to the appendix, and state in the human-evaluation section of the main text how far the results extend.

Reviewer_Si64OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles interpretability in image aesthetic assessment (IAA) and proposes FaTeX, whose central claim is faithful textual explanation: the prediction is structurally written as a weighted sum of named-concept contributions (Eq. 1), m(x)=ρ(x)∑tp(t∣x),dθ(x,t)m(x)=\rho(x)\sum_t p(t\mid x),d_\theta(x,t), so each concept's contribution sums exactly to the output — unlike post-hoc explanations such as Grad-CAM or post-hoc CBMs.

A concept pool is built offline per dataset in three steps: Gemini-3.5 Flash generates antonymic prompt pairs from three sources (attribute seeds, general, and per-content-level with sample images); a frozen CLIP scores each pair and a content-controlled partial rank correlation (Eq. 2) decides eligibility; OMP then prunes paraphrases. At prediction time, Sec. 3.3 gives a frozen CLIP-IQA-style response dθd_\theta (only a per-concept gain is learned), and Sec. 3.4 learns the per-image concept weights p(t∣x)p(t\mid x) via sparsemax applicability gating plus a learned allocation. Supervision is the human mean aesthetic ratings of six public datasets; the only trainable parameters are the weighting path, the per-concept gain, and the magnitude head, with everything else anchored to frozen CLIP.

Experiments cover six datasets (LAPIS, APDDv2, PAMELA, AADB, TAD66K, PARA) under two backbones (CLIP-B/16, UniQA-B/16), measured by SRCC/PLCC against six baselines including a non-interpretable MLP head, NIMA, and recent interpretable methods (PPA-T, C2J). FaTeX is competitive but does not match the best baseline on any dataset: its "Δ to column best" is negative throughout (e.g., CLIP-B/16: −.017 to −.058 SRCC). The paper's case therefore rests on pairing near-competitive accuracy with faithful explanations.

Critical strengths
  1. Faithfulness is a real design property. The prediction is constrained to be an additive sum of named-concept contributions, so the explanation equals the sum of contributions rather than approximating it. Confining learnable capacity to the weighting while anchoring the response to frozen CLIP is a disciplined, well-judged trade-off for explanation faithfulness.
  2. The concept-pool construction is engineered carefully. The A/B/C division gives reasonable coverage; Step 2's content-controlled partial correlation correctly removes the "content → rating" confound that IAA work often ignores; the evaluative/descriptive distinction with one-/two-sided tests and automatic polarity swapping is thoughtful; and Step 3's OMP pruning avoids piling synonyms into explanations. The appendix gives the verbatim prompt, slot values, and de-duplication rules in full, so reproducibility is good.
Critical weaknesses
  1. Accuracy is a net loss against SOTA, and the paper underplays it. In Table 1 the "Δ to column best" row is negative on all six datasets (with CLIP-B/16: LAPIS −.017, APDDv2 −.042, PAMELA −.033, AADB −.058, TAD66K −.031, PARA −.026). Interpretability is bought with a consistent accuracy cost, and on APDDv2/AADB the gap is not small. The paper sells faithfulness but never quantifies how much accuracy a faithful explanation costs; the authors should confront this trade-off directly rather than leaving readers to read the Δ row themselves.

  2. The necessity of several components is questionable. By the authors' own Table S5, removing the gate barely changes SRCC (+.001/+.006) and removing ρ\rho is "inconsistent." So the Stage-1 applicability gate and the magnitude ρ\rho appear to serve explanation quality rather than prediction. The authors do argue this (the gate affects the cross-level share in the top-3), but it means several core components rest entirely on explanation quality while contributing almost nothing to prediction, and this needs stronger evidence that they genuinely improve the explanations people read.

  3. Heavy reliance on Gemini / CLIP raises robustness and reproducibility concerns. (a) The pool is generated by Gemini-3.5 Flash at temperature 1.0 — high variance and dependent on a closed API; the paper reports no sensitivity of the pool to the generation seed. (b) AADB's content tags are assigned by the same MLLM; despite calibration (acc .851), κ = .83 still implies ~15% mislabeling that enters both the Eq. 2 covariate and the Sec. 3.4 gate, un-ablated. (c) The response side is anchored to CLIP-IQA, so the ceiling is capped by the backbone (the consistent UniQA > CLIP gap supports this).

Questions for authors
  1. Please address the accuracy–interpretability trade-off head-on: given a consistent deficit to column-best across all six datasets, how is the benefit of faithfulness quantified? Is there a stronger-backbone configuration that sacrifices only a little accuracy?
  2. How sensitive is the concept pool to the Gemini generation seed? What is the overlap of T\mathcal{T} across repeated generations, and the resulting variance in downstream SRCC?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Si64 (Part 1/2)

Thank you for reviewing our submission. Thanks to your comments, the manuscript now addresses the accuracy cost directly and reports the sensitivity to the generation seed of the concept pool, which makes the limitations and reproducibility of the method clearer. We respond to each point below.

  1. Please address the accuracy–interpretability trade-off head-on: given a consistent deficit to column-best across all six datasets, how is the benefit of faithfulness quantified? Is there a stronger-backbone configuration that sacrifices only a little accuracy?

[Response] We agree with this point. We read it as saying that our Discussion and Limitations do not discuss the trade-off sufficiently. Below we answer where the cost comes from, whether a stronger configuration exists, and what the cost buys.

In short, the accuracy cost comes from reading the prediction out as a sum of named concept contributions, not from the components that shape the explanation or from the narrow fine-tuning, and a stronger backbone does not close the gap. In exchange, compared with PPA-T, the closest decomposable model, FaTeX gives up .007–.026 SRCC on the five datasets other than APDDv2, while its explanations are preferred in both the human and the LLM evaluations and single out the image.

Below, paired values are given as LAPIS / PARA.

Where the cost comes from. The components that shape the explanation do not lower accuracy beyond the range of SRCC across three seeds (.007 / .003). The only exception is PPA's fixed vocabulary on LAPIS, where our selected vocabulary is the more accurate one. What these components change is the explanation. In the table, ΔSRCC is the model retrained without the component minus FaTeX; a negative value means lower accuracy.

Component removed (Tables S5, S7)ΔSRCCExplanation (FaTeX → without it)
Applicability gate+.001 / +.006Other-level share of the top three: .174 → .595 / .136 → .707
Tag term−.006 / .000Own-level share of the top three: .250 → .141 / .324 → .212
Content control−.003 / +.002On LAPIS, concepts that only respond to what is depicted enter the explanations of 48% of test images
Rating-selected vocabulary (replaced by PPA's fixed 260 pairs)−.013 / +.004Distinct concepts in the top three: 100 → 61 / 123 → 77

Widening the trainable scope does not close the gap either. Fine-tuning the last four blocks in full, the scope of MLP and PPA-T, gives an SRCC of .833 / .887, lower than the LayerNorm-only FaTeX (.846 / .891) (one run, learning rate searched as for the main model). The cost therefore lies in reading the prediction out as a sum of named concept contributions, not in the gating or in the narrow fine-tuning.

Stronger backbone. UniQA-B/16, pre-trained on quality and aesthetics text, is already in Table 1. It helps FaTeX on the multi-genre datasets, but it helps the baselines as much, so the gap remains (AADB: FaTeX .654, best baseline .712). Because the concept responses are read from a frozen encoder, the backbone sets the ceiling, as the reviewer notes in W3(c). The configurations with the smallest cost are LAPIS with CLIP-B/16 (.017 below the column best) and PARA with UniQA-B/16 (.026 below).

What the cost buys. The most direct comparison is PPA-T, the other method whose prediction decomposes into concepts. On the five datasets other than APDDv2, PPA-T is more accurate by .007 to .026 SRCC; FaTeX leads on APDDv2 (+.071). In return, on LAPIS and PARA, where we ran the human evaluation:

  • Raters preferred FaTeX's explanations over PPA-T's on all three questions on both LAPIS and PARA (overall quality: 71.2% on LAPIS, 65.0% on PARA; every 95% CI excludes 50%).
  • From FaTeX's explanations, raters identified the described image 37.5% / 36.2% of the time, while PPA-T stays at the 25% chance level (22.9% / 25.0%).
  • Among the 40 images of a rating band, PPA-T has bands with only 1–3 distinct explanations, while FaTeX produces 35–39.

On the other four datasets (AADB, APDDv2, PAMELA and TAD66K), which the human evaluation did not cover, we ran the same two tasks with the LLM judge, as described in our reply to Reviewer o1fi. The advantage over PPA-T holds there as well: FaTeX is preferred on overall quality in 62.9% to 78.2% of Task 1 comparisons, and its Task 2 accuracy exceeds PPA-T's by 14.7 to 27.0 points, with every 95% CI excluding chance.

[Action] Using the additional page allowed in the revision, we make the following changes:

  • We add a paragraph to the Discussion that states the accuracy cost, where it comes from, and what it buys.
  • We correct the tone of the Discussion. For example, we remove "applications that need explanations thus have little reason to settle for a black box".
AuthorsOpenReview ↗

Response to Reviewer Si64 (Part 2/2)

  1. How sensitive is the concept pool to the Gemini generation seed? What is the overlap of T across repeated generations, and the resulting variance in downstream SRCC?

[Response] In short, the wording of the pool changes substantially from one generation to the next, but downstream accuracy barely changes. Independent generations share only 13% / 15% of their T0∗T^{\ast}_0 pairs and 3% / 2% of their Tg∗T^{\ast}_g pairs verbatim, while the SD of test SRCC is .006 / .002 and the mean differs from the original pool by −.007 / +.003 (LAPIS / PARA).

We regenerated the pool three more times from scratch. All three sources (A, B, C) were regenerated with Gemini seeds 42, 43 and 44, with the example images held fixed. Because gemini-3.5-flash is no longer available through the Gemini API, these three generations come from gemini-3.6-flash. We retrained the model on LAPIS (CLIP-B/16) and PARA (UniQA-B/16) with the main model's learning rate, seed and epochs, as in Table S7. Overlaps are averaged over the three pairs of generations.

LAPISPARA
T0∗T^{\ast}_0: share of identical pairs (Jaccard)13%15%
Tg∗T^{\ast}_g: same3%2%
T0∗T^{\ast}_0: share with a near-equivalent (|corr| ≥ .7) in the other generation54%59%
Tg∗T^{\ast}_g: same (|corr| ≥ .7)31%34%
Test SRCC, pools regenerated with gemini-3.6-flash.844 / .833 / .840 (SD .006).896 / .891 / .893 (SD .002)
Reference: test SRCC, original pool with gemini-3.5-flash.846 / .846 / .839 (SD .004).891 / .889 / .892 (SD .001)

The three regenerated-pool values correspond to Gemini generation seeds 42 / 43 / 44, and the three reference values to training seeds 42 / 43 / 44.

Wording and concepts. The wording of the pool is not unique. In particular, Tg∗T^{\ast}_g, which is written per content level, is almost entirely replaced. The concepts are not entirely different, however. In the collection-wide T0∗T^{\ast}_0, independent generations share 13% (LAPIS) and 15% (PARA) of their pairs verbatim. Examples are Atmospheric/Sterile and Intentional/Accidental on LAPIS, and Vibrant/Dull, Serene/Chaotic and Organic/Geometric on PARA. About half of the remaining T0∗T^{\ast}_0 concepts have a counterpart in the other generation that responds almost identically on the same images.

Downstream accuracy. SRCC varies little across the regenerated pools (SD .006 on LAPIS and .002 on PARA), and their mean differs little from that of the original pool's model (−.007 on LAPIS, +.003 on PARA). Accuracy therefore does not rest on one particular wording, and selecting candidates by the same rules recovers it reliably.

Change of generator. These regenerations also include a change of generator from the original pool (3.5-flash). Even so, the original pool overlaps the three 3.6 pools only slightly less than they overlap each other (share of identical T0∗T^{\ast}_0 pairs 10% / 7% vs 13% / 15%), and the SRCC difference is small (.846 / .891 for the original pool vs a mean of .839 / .894 for the three 3.6 pools).

Reproducibility. That LLM generations cannot be reproduced exactly is a challenge shared by all work that relies on proprietary LLMs; Label-Free CBM (Oikarinen et al., 2023), for example, also generates its concept sets with GPT-3 (text-davinci-002) through the OpenAI API, acknowledges that this generation is stochastic, and reports that regenerating the set leaves accuracy nearly unchanged while the concepts used can differ considerably. In our method, the pool needs to be generated only once, offline, before training, and we release all generated pools and candidates. Pair construction, selection and OMP (Steps 2–3) are deterministic. The results are therefore reproducible from the released pools, at least as machine-learning experiments.

[Action]

  • We add this table to Appendix, together with the earlier regeneration of source C alone and the generation by a different LLM.
  • We state in Appendix that a pool's wording is not unique and that the words in the explanations change between generations.
  • We state in the main text that the released pools and candidate sets are fixed artifacts.

[References]

  • Oikarinen et al. Label-Free Concept Bottleneck Models. ICLR, 2023.
AuthorsOpenReview ↗

Follow-up on Our Response Before the Discussion Period Ends

Dear Reviewer,

Thank you again for your time and thoughtful feedback. We understand that this is a busy period.

As the discussion period is drawing to a close, we would like to respectfully follow up on our previous response. We would be very grateful if you could let us know whether our responses have adequately addressed your concerns, or if there are any remaining questions that we could further clarify.

We sincerely appreciate your time and consideration.

Best regards,

The Authors

Reviewer_LQ6ROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FaTeX, a model for image aesthetic assessment aims for faithful textual explanations. FaTeX prediction is an exact sum of contributions from named antonymic prompt pairs, scored by a frozen CLIP and weighted per image followed by a learned allocation. The concept pool is generated by an LLM and selected per dataset by rank correlation with the rating, controlling for the content tag. On six IAA datasets FaTeX demonstrated the best performance, although some of the benchmark results are not shown. In human subjective and LLM-as-a-judge studies on LAPIS and PARA, the explanations from FaTeX are preferred over, and are more discriminative than those of PPA-T and C2J.

Critical strengths

The problem framing of the paper is well-chosen. It identifies why the standard LLM+CLIP fails for aesthetics, such as concepts collapse into paraphrases of "good image", and content-tracking words leak into the explanation.

The accuracy cost of interpretability is quantified across six datasets with matched training budgets. The human study is carefully designed and its results agree in direction with the LLM judge, which makes the experiments convincing.

The ablations show that removing the weight components would change the explanations rather than the accuracy, which again emphasizes the main idea of the paper that the proposed model is more focused on interpretability rather than only SRCC/PLCC.

Critical weaknesses

Currently the paper does not include cross-dataset generalization. The pool is built and the model is trained per dataset. How well does a trained FaTeX, together with its pool, transfer to another dataset?

The completeness of Table 1 could be substantially improved by supplying the missing data in the currently empty cells. In instances where specific values are unobtainable, the authors should provide a detailed methodological justification or clarification for their omission.

Questions for authors

The human pairwise evaluation seems to measure relative preference, rather than whether explanations are correct.

The human evaluation images are easy cases. They are restricted to images on which every compared method predicts within 0.5 SD of the ground truth. These are relatively easy cases. It remains unclear whether the conclusions hold when the model's prediction is wrong, which is arguably where explanations are most useful.

6 comments
AuthorsOpenReview ↗

Response to Reviewer LQ6R (Part 1/4)

Thank you for reviewing our submission. Thanks to these comments, we checked whether our conclusions hold on unfiltered images and on images where the model is wrong, which clarifies the scope of our conclusions. We respond to each point below. For the question on relative preference versus correctness, an additional human evaluation is still in progress; we will post our response to it as soon as it is complete.

Currently the paper does not include cross-dataset generalization. The pool is built and the model is trained per dataset. How well does a trained FaTeX, together with its pool, transfer to another dataset?

[Response] In short, FaTeX transfers well to PAMELA and AADB, keeping 95–97% of the SRCC of a model trained on the target, a drop smaller than that of PPA-T or MLP. No method transfers to APDDv2 or TAD66K (50–67%). This likely reflects differently defined ratings.

We measured this directly. Without retraining, we applied FaTeX trained on one dataset, together with its pool, to another: LAPIS (single genre) to the other painting datasets, PAMELA and APDDv2, and PARA (multi-genre) to the other photo datasets, AADB and TAD66K. The rating scales differ, so we compare SRCC. Under the same conditions we also scored PPA-T, whose fixed vocabulary is shared by all datasets, and the non-interpretable MLP.

Trained on → tested onFaTeXPPA-TMLPFaTeX trained on the target
LAPIS → PAMELA (CLIP).749.725.718.773
LAPIS → APDDv2 (CLIP).476.439.427.808
PARA → AADB (UniQA).621.628.637.654
PARA → TAD66K (UniQA).299.264.286.449

SRCC on the target's test split. For each pair, the backbone is the one with the higher median on both source and target. Values are from the run with the median SRCC on the source's test split (as in Table 1). Last column: FaTeX trained on the target itself (Table 1).

The results split by target.

  • Transfer to PAMELA and AADB works well. FaTeX keeps 95–97% of the SRCC of a model trained on the target, a gap of .02–.03. This gap is smaller than that of PPA-T (.06 on PAMELA, .05 on AADB) or MLP (.09 on PAMELA, .07 on AADB).
  • No method transfers to APDDv2 or TAD66K, all reaching 50–67% of the model trained on the target. APDDv2 is scored by art experts against a table of reference images prepared for each medium, and about a quarter of its works are by art students; TAD66K is rated independently within each theme. Since the rating itself is defined differently from the source, part of it cannot be learned from another dataset's mean ratings.

Generalization across datasets is an important property in practice. It is not a main focus of this work, but we see it as a promising direction for future work. Because both the centring and the concept selection in FaTeX are designed per dataset, refining them may improve transfer; confirming this requires further study.

[Action] We will add this table to the appendix. In the discussion (Sec. 5), we will state that transfer holds between datasets rated in a similar way but does not extend to datasets whose rating is defined differently, together with directions for future work.

AuthorsOpenReview ↗

Response to Reviewer LQ6R (Part 2/4)

The completeness of Table 1 could be substantially improved by supplying the missing data in the currently empty cells. In instances where specific values are unobtainable, the authors should provide a detailed methodological justification or clarification for their omission.

[Response] Thank you for pointing this out. In short, every empty cell in Table 1 belongs to NIMA or C2J, and each is left out for a reason specific to the method. NIMA needs a per-image rating distribution, which the four datasets in question do not provide. C2J is limited to the datasets for which its original paper defines concept sets (AADB, PARA, LAPIS), to avoid choosing concepts arbitrarily elsewhere.

NIMA (APDDv2, PAMELA, AADB, TAD66K). NIMA does not regress the mean score. It predicts a distribution over the discrete levels of the rating scale and is trained with the EMD loss against each image's rating histogram. So it needs either per-rater ratings or a histogram over discrete levels for every image.

  • APDDv2, AADB and TAD66K release only per-image means. There are no per-rater records and no histograms.
  • PAMELA's ratings come from a continuous slider, so there are no discrete levels to build a histogram over.

We therefore report NIMA only on LAPIS and PARA, where the target distribution exists. Appendix D gives the reason for each dataset.

C2J (APDDv2, PAMELA, TAD66K). C2J's concepts are not fixed. Each dataset needs a choice of which annotations become concepts and which images demonstrate each concept. AADB, PARA and LAPIS are also used in the original C2J paper, so we can follow its choice of concept sets directly.

For the other datasets, there is no concept set the original authors chose, and building one would depend on decisions we would have to make ourselves. In particular:

  • TAD66K has no attribute annotations at all.
  • APDDv2 has ten attribute columns, but which ones are defined depends on the medium (0–62% missing per column). We would have to decide which columns become concepts and how to treat images where a column is missing.

To exclude arbitrary choices and keep the comparison faithful to the original method, we restrict C2J to the datasets and concept sets defined in the original paper.

[Action] We will add the reasons for this scope to the appendix.

AuthorsOpenReview ↗

Response to Reviewer LQ6R (Part 3/4)

The human evaluation images are easy cases. They are restricted to images on which every compared method predicts within 0.5 SD of the ground truth. These are relatively easy cases. It remains unclear whether the conclusions hold when the model's prediction is wrong, which is arguably where explanations are most useful.

[Response] In short, in a new human evaluation without the filter and with the predicted band, raters identified the image from FaTeX's explanations (Task 2) above chance even on images the model got wrong (43.3% on LAPIS, 41.7% on PARA), no lower than on images it predicted well. Preference (Task 1) held on LAPIS, where FaTeX was chosen about as often on images the model got wrong as on the others, but narrowed on the PARA images the model got wrong, where the difference from PPA-T was no longer clear.

As the reviewer notes, the human study used only images on which every model predicts within 0.5 test-set SD of the target. The filter keeps differences in predictive accuracy out of the comparison of explanations, but it limits our conclusions to images on which the models are roughly correct. The numbers and fractions that pass are:

DatasetTestPass (all models)Pass rate by band (low / average / high)
LAPIS2,345843 (35.9%)32.0% / 40.8% / 35.0%
PARA3,0001,577 (52.6%)43.2% / 69.4% / 45.4%

The original study cannot answer this question: it contains no image on which the model is wrong. Also, the template's level word was the ground-truth band rather than the model's prediction. This kept the level word identical across the two explanations being compared, so that raters compared only the concepts; with predicted bands, the level word would change whenever the methods' predictions differ. We therefore ran an additional human study to measure two things:

  • (a) Are the explanations still useful on images the model gets wrong?
  • (b) Do the original conclusions hold as the explanations would be used (no filter, level word from the model's predicted band)?

Everything else follows the original study, except:

  • Images: without the filter, we drew 60 images at random from those FaTeX misses (|error| > 0.5 SD) and 60 from those it does not. Missed images are oversampled relative to their share of the test set (16.3% on PARA, 35.7% on LAPIS), so that (a) is answered with enough images. For (b), we weight the two strata by their shares of the test set.
  • Level word: each model's predicted band instead of the ground-truth band.

As this is a follow-up study, we reduced the raters per item from two to one, which widens the confidence intervals.

LAPIS: originalLAPIS: random (weighted)LAPIS: model wrongPARA: originalPARA: random (weighted)PARA: model wrong
Task 1: FaTeX preferred (Q0, vs w/o Tg∗T^{\ast}_g)63.7%63.5%66.7%65.4%70.3%55.0%
Task 1: same (vs PPA-T)71.2%77.9%80.0%65.0%56.4%46.7%
Task 1: same (vs C2J)56.2%58.9%60.0%69.6%72.5%68.3%
Task 2: FaTeX accuracy (chance 25%)37.5%45.5%43.3%36.2%37.5%41.7%

"Model wrong": the 60 images FaTeX misses by more than 0.5 SD. "Original": the submitted study.

(a) Images on which the model is wrong. In Task 2, on the 60 images the model misses, raters identified the correct image from FaTeX's explanation 43.3% of the time on LAPIS (95% CI 31.7–56.7%) and 41.7% on PARA (30.0–55.0%), above chance (25%) and no lower than on the 60 images it predicts well. In Task 1, LAPIS raters preferred FaTeX about as often on missed images as on the others, whereas on PARA FaTeX's advantage narrowed on missed images, to 46.7% against PPA-T (58.3% on well-predicted images).

(b) Unfiltered test images. In the weighted estimates, Task 2 accuracy is at or above the original setting. On LAPIS, the 95% CIs of the differences from all three methods exclude zero; on PARA, the differences from C2J and PPA-T have the same sign as before, with CIs that just include zero. In Task 1, LAPIS is close to the original setting. On PARA, FaTeX is preferred at or above the original rate over w/o Tg∗T^{\ast}_g and C2J, but its preference over PPA-T falls to 56.4% from 65.0%, with a CI that includes 50%.

With one rater per item, the confidence intervals are wider than in the original setting, and more comparisons have intervals that include chance or zero. Even so, FaTeX keeps the same direction of advantage as in the original setting in almost every comparison. The exception is Task 1 against PPA-T on PARA, where the preference reverses on images the model gets wrong (46.7%).

[Action] We will add the pass-rate table to the appendix, together with the unfiltered, predicted-band human evaluation and its results on the images the model misses. In the human-evaluation section of the main text, we will state how far the results extend.

Reviewer_o1fiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FaTeX, an interpretable framework for Image Aesthetic Assessment (IAA), which predicts human aesthetic ratings and provides textual explanations. Their main motivation is that existing IAA models achieve good predictive accuracy, but their explanations may not faithfully reflect the model's predictions or capture image-specific aesthetic properties.

The main distinctions made by this work are: i) FaTeX represents predicted aesthetic ratings through a weighted sum of signed concept contributions, allowing the prediction to be decomposed into interpretable concepts; ii) FaTeX uses LLMs and CLIP to construct aesthetic concept vocabularies, filters content-driven correlations and redundant concepts, and selects relevant concepts for each image without requiring concept-level annotations; and lastly, iii) the paper evaluates three properties: faithfulness, fit to the image, and specificity, through human evaluation, LLM judgments, and concept-deletion experiments.

Their main contribution is improving the interpretability of aesthetic predictions while maintaining reasonable predictive performance. However, as discussed in the XAI literature and elaborated in our weaknesses, the semantic validity of signed concept contributions and their robustness to design choices remain insufficiently evaluated.

Critical strengths
  1. Well motivated problem. The paper identifies a meaningful challenge in applying concept-based interpretability to the subjective image aesthetic assessment problem. This sufficiently motivates their proposed approach.

  2. The related work section is well organized and provides a clear discussion of existing approaches and their limitations.

  3. FaTeX achieves an average SRCC (Spearman rank correlation) gap of only 0.035 from the best-performing methods across six datasets, demonstrating a reasonable trade-off between predictive accuracy and interpretability.

  4. Their evaluation goes beyond subjective preferences by including an image-identification task. The FaTeX achieves above-chance human accuracy on both datasets studied (37.5% on LAPIS and 36.2% on PARA).

  5. Concept-deletion experiments show that removing high-contribution concepts substantially degrades predictive performance, while ablations demonstrate the benefits of content-specific vocabularies and applicability gating for explanation quality.

  6. The supplementary material includes well-discussed visual case studies that illustrate the differences in explanations across methods and help a reader better understand the strengths and limitations of FaTeX.

Critical weaknesses

W1. Novelty positioning.

Since faithfulness, image relevance/groundedness, and specificity are well-established explanation criteria, the novelty claim in Contribution (1) can be more clearly distinguished from prior work. It would also be useful to clarify that exact arithmetic decomposition guarantees computational faithfulness, whereas image fit and specificity require empirical validation rather than being guaranteed by the formulation.

W2 (Moderate): Missing citations in Introduction

The related work already covers Jacovi & Goldberg (2020) and Oikarinen et al. (2023), but these references would also help ground the introduction where faithfulness and annotation-free concept bottlenecks are first discussed. Importantly, the three explanation criteria should be connected to existing research, particularly He et al. (ACL 2026) on visual fidelity and contrastiveness. Aesthetic Perception Prompting (ICASSP 2025) is also worth acknowledging as prior work on scene-aware textual explanations for IAA. While the related work clarifies FaTeX's methodological distinctions, the introduction could better establish what remains unresolved and avoid presenting established explanation criteria as novel.

W3. Validity of Signed Concept Response.

In Section 3.3, concept responses are centered relative to the training-set mean. Therefore, a negative contribution may indicate a below-average CLIP response rather than presence of the negative aesthetic attribute. How often do the reported positive and negative contributions agree with human judgments about which attribute is actually present in the image? The current human evaluation assesses overall explanation quality but does not directly validate the polarity of individual concept responses.

W4. Sensitivity/Robustness of Explanations.

Prior works in explainable AI always emphasized that meaningful explanations should be stable under small perturbations and reasonable modeling choices (Alvarez-Melis & Jaakkola, 2018; Adebayo et al., 2018). As we stated in W3, concept contributions depend on mean-centered CLIP responses and learned gains, yet their sensitivity is not evaluated anywhere. How stable are the top-ranked concepts and their relative contributions under alternative centering choices or variations in learned gains, particularly when predictive performance remains comparable?


Minor-W1:

FaTeX underperforms the MLP baseline across all six datasets, but explanation quality is evaluated only on LAPIS and PARA. To what extent do the stated interpretability benefits generalize to the remaining datasets, particularly those with larger predictive losses?

Minor-W2:

FaTeX outperforms PPA-T by 0.07 SRCC on APDDv2 (with CLIP-B/16) in Table 1, unlike the other five datasets. Could the authors briefly discuss possible factors behind this dataset-specific improvement?


Citations:

Jacovi & Goldberg et al., "Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness?", ACL 2020.

He et al., "Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations", ACL 2026.

Oikarinen et al., "Label-Free Concept Bottleneck Models" , ICLR 2023.

L. Wang et al., "Aesthetic Perception Prompting for Interpretable Image Aesthetics Assessment with MLLMs", ICASSP 2025.

Alvarez-Melis & Jaakkola et al., "Towards Robust Interpretability with Self-Explaining Neural Networks", NeurIPS 2018.

Adebayo & Gilmer., "Sanity Checks for Saliency Maps", NeurIPS 2018.

Questions for authors

Q1. GIAA is used in Fig 1 caption without being explicitly defined. Please define it at first mention and clarify how the individual rater scores were aggregated for LAPIS and PAMELA.

Q2. Could the authors better ground the three explanation criteria in prior literature and clarify how Contribution 1 differs from these established criteria (see W1 and W2)?

Q3. As discussed in W3, how often does the polarity of signed concept contributions agree with human judgments of the actual visual attributes?

Q4. As discussed in W4, how stable are the top-ranked concepts and their contributions under alternative centering choices or variations in learned gains, particularly when predictive performance remains comparable?

Q5. Can authors address minor weaknesses discussed in Minor-W1 and Minor-W2?


Q1 and Q2 are clarification questions but Q3 , Q4 are concerns that need additional evidence and discussions.

10 comments
AuthorsOpenReview ↗

Response to Reviewer o1fi (Part 1/5)

We thank the reviewer for expert comments grounded in interpretability research and for listing specific references that help position this work. Thanks to these comments, the manuscript now states the scope and definition of faithfulness and its relation to prior work more clearly, and it adds tests of the stability of the explanations. We respond to each point below. For Q3, an additional human evaluation is still in progress; we will post our response to Q3 as soon as it is complete.

Q1. GIAA is used in Fig 1 caption without being explicitly defined. Please define it at first mention and clarify how the individual rater scores were aggregated for LAPIS and PAMELA.

[Response] You are right that GIAA and the aggregation of ratings should be made clear. We will define GIAA (general image aesthetic assessment) at its first mention.

For both LAPIS and PAMELA, the target is the unweighted mean of all individual ratings of an image.

  • LAPIS: we use the per-image mean released with the official splits (13–134 ratings per image, 24.2 on average).
  • PAMELA: only per-rater records are released, so we average the raw slider values over all raters of an image (7–17 ratings, 13.3 on average).

[Action] We define GIAA where the task is formulated (Sec. 3.1), and we state the aggregation for each dataset in Appendix D.

AuthorsOpenReview ↗

Response to Reviewer o1fi (Part 2/5)

Q2. Could the authors better ground the three explanation criteria in prior literature and clarify how Contribution 1 differs from these established criteria (see W1 and W2)?

[Response] We thank the reviewer for pointing us to these works.

Our main point is that proposing the three criteria is not our contribution. Our contribution is an interpretable model suited to aesthetic assessment and the design of experiments that test its explanations. In particular, Task 2 is designed so that people can judge whether an explanation is specific to its image, in a task where explanations have no ground truth.

With this positioning, our view of the three criteria, based on the works the reviewer suggested, is as follows.

  • Faithfulness. Jacovi & Goldberg (2020) argue that faithfulness and plausibility are separate criteria, and that faithfulness should not be evaluated through human judgment. They also point out that a strictly faithful interpretation is a "unicorn", and that faithfulness should not be judged as a binary property. Following them, we now treat faithfulness within a more appropriate scope and under a clearer definition: the explanation decomposes the prediction, which holds by construction and is faithfulness in the computational sense only. We do not offer the human evaluation as evidence of faithfulness. Whether the concept names capture the reason for a prediction is a further question, which we address under Q3.
  • Fit to the image and specificity. Our two criteria (fit to the image and specificity) correspond, respectively, to Visual Fidelity and Contrastiveness of He et al. (ACL 2026). He et al. score these automatically with GPT-4o and an entailment model, which presupposes that the statements in an explanation are clearly and objectively right or wrong. In aesthetic assessment, whether a statement such as "a harmonious composition" fits an image has no objective answer and is decided by human subjective judgment, so their automatic scores do not apply directly. We therefore asked human raters directly whether an explanation fits its image and whether it singles out its image among others (Task 2).

Two of the works the reviewer suggested also bear on how our method is positioned.

  • Concept bottleneck models. Our method is inspired by Label-Free CBM (Oikarinen et al., 2023), which obtains concepts from an LLM and CLIP without concept annotations. This should have been stated in the introduction as well as in the related work.
  • Textual explanations in IAA. Aesthetic Perception Prompting (Wang et al., 2025) gives an MLLM attribute-wise criteria and reference images of the same scene, and has it generate a score and an explanation. Its view that the relevant criteria change with the scene is shared by our design, which builds a vocabulary per content level. Its explanation, however, is generated alongside the score rather than decomposing it, so it is not faithful in our sense.

[Action] We will cite Jacovi & Goldberg and Oikarinen et al. in the introduction where faithfulness and annotation-free concept bottlenecks first appear, and briefly relate fit to the image and specificity to He et al. The related work will explain why their scores do not apply directly. Throughout the manuscript, we will use faithfulness with the scope and definition above. We will also cite Wang et al. (2025) in the introduction and the related work.

[References]

  • Jacovi and Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? ACL, 2020.
  • He et al. Believing without Seeing: Quality Scores for Contextualizing Vision-Language Model Explanations. ACL, 2026.
  • Oikarinen et al. Label-Free Concept Bottleneck Models. ICLR, 2023.
  • Wang et al. Aesthetic Perception Prompting for Interpretable Image Aesthetics Assessment with MLLMs. ICASSP, 2025.
AuthorsOpenReview ↗

Response to Reviewer o1fi (Part 3/5)

Q4. As discussed in W4, how stable are the top-ranked concepts and their contributions under alternative centering choices or variations in learned gains, particularly when predictive performance remains comparable?

[Response] Sensitivity is an important property that the submission did not evaluate. We have now measured it, separating perturbations of a trained model from retraining with another seed. The cells are LAPIS×CLIP-B/16 and PARA×UniQA-B/16, on all test images, averaged over three runs.

In short, the relative contributions of concepts are stable when predictive performance remains comparable. The per-image rank correlation of contributions is at least .95 under changes in centering and gains, and .84 / .88 even after retraining with another seed, well above that between different images of the same content level (.50 / .65).

ChangeΔSRCCRank correlation of ∣ct∣\lvert c_t \rvertTop-3 agreement
μ\mu → training median−.001 / +.002.984 / .990.93 / .96
μ\mu → mean of 4,000 random images+.000 / −.000.993 / .997.97 / .99
γ\gamma × exp⁡(N(0,.252))\exp(\mathcal{N}(0, .25^2))−.001 / −.001.986 / .994.82 / .90
γ\gamma from another seed−.004 / −.003.951 / .957.77 / .81
Retraining with another seed.005 / .002.841 / .875.49 / .51
Reference: another image of the same level, same model—.497 / .649.23 / .21

Values are LAPIS / PARA. The rank correlation is the Spearman correlation of ∣ct∣\lvert c_t \rvert over the applicable concepts of an image (86 / 105 on average), averaged over all images. Top-3 agreement is the share of an image's top three concepts in common. In the retraining row, ΔSRCC is the mean absolute difference in test SRCC between pairs of runs. For the random-images row, we randomly sampled 4,000 training images (about 49% of LAPIS and 18% of PARA) and recomputed the mean. Sampling and noise use 10 draws per run. The first four rows involve no retraining, with everything else held fixed.

The seed affects the initial weights and the learning-rate search. Retraining with another seed preserves the rough ordering of importance in the explanations. The rank correlation of ∣ct∣\lvert c_t \rvert is .84 / .88, well above that between different images of the same level (.50 / .65). In addition, 81% / 83% of the top three concepts stay within the top 10 of another seed (out of 86 / 105 applicable concepts per image on average; 44% / 40% for another image of the same level). Partial swaps among the top concepts can still occur: about half of the top three are shared (.49 / .51). The top three are also fairly robust to the other, small perturbations (agreement .77 to .99). That concepts with similar contributions swap order easily may reflect the nature of aesthetic assessment itself, in which it is hard to settle uniquely which reason matters more. Confirming this, however, requires further study.

All of these perturbations change SRCC only slightly (at most .005). For the changes we examined, these results therefore answer your question of how stable the concepts are when predictive performance remains comparable.

The human evaluation concerns each method's representative run. To check whether swaps in the top three affect the results, we repeated the LLM evaluation with FaTeX's explanations from the other two seeds, keeping the baselines' explanations fixed. The LLM's conclusions did not weaken under the other seeds. FaTeX stayed on the same side in all 18 Task 1 comparisons (2 datasets × 3 baselines × Q0–Q2), and none fell below the 95% CI of the representative run. Task 2 accuracy was close to the representative run on LAPIS (52.1% and 52.9% against 53.8%) and higher on PARA (48.8% and 50.0% against 40.8%).

[Action] We added this analysis, with the table above, to the appendix, and added to the limitations that the top concepts can partly change with the seed.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised manuscript uploaded

We sincerely thank all reviewers for their time and their careful, constructive comments. They have substantially improved the manuscript. We have uploaded a revised version that makes the changes promised in our responses, using the additional page allowed for the revision.

All changes are highlighted in blue in the revised PDF, so that they can be checked against our responses.

The additional human studies reported in the revision fall within the scope of the ethics approval that covered the original human evaluation, and they followed the same procedures for consent, payment and data handling. These are the human study without the filter, using the predicted band, and the human study of concept polarity. Upon acceptance, we will release their materials and anonymized responses, as we have done for the original human evaluation.

We would be happy to answer any further questions.

AuthorsOpenReview ↗

Summary of the Discussion for the Area Chair

Dear Area Chair,

Thank you for handling our submission. As the discussion period closes, we give you a single overview: we summarize where each reviewer stands, how the main concerns were addressed in the revised PDF, and which limitations remain.

We are grateful to all four reviewers. The additional experiments their comments prompted (on information leakage, concept polarity, images the model gets wrong, and seed sensitivity) clarified the scope of our claims: in what sense the explanations are faithful, and on which images the human evidence holds.

1. Reviewer status

ReviewerInitialCurrentStatus
o1fi23Raised after discussion; "most of my concerns have been substantially addressed"
LQ6R33"All my concerns have been well-addressed"
w8Cr33No reply
Si6422No reply
Avg.2.502.75

2. Main concerns and how we addressed them

(1) Leakage and the scope of faithfulness (w8Cr Q1–Q2; o1fi W1). Some leakage is expected when the concept set does not carry all the rating-relevant information in the image (Espinosa Zarlenga, 2026), as is likely for aesthetic vocabulary. What the submission lacked was a measurement of how much leakage there is. A leakage test on all six datasets now reports it: with random words in place of the concepts, the weight path still recovers most of the rating (+.071 to +.209 SRCC), while the real concepts carry rating information of their own (+.081 to +.169 over random words without the path). We therefore claim faithfulness only computationally: the prediction is an exact sum of named contributions (Jacovi & Goldberg, 2020). That concepts capture what they name is shown separately: weakening a named attribute in the image turned its contribution negative in 77–100% of images (a flip control did not); see also (2). The three-concept text keeps the prediction's direction (92%) and ranking (SRCC .94 / .92), not its magnitude. (App. M, N)

(2) Correct, not only preferred? (LQ6R; o1fi W3). In Task 2, raters picked the described image among four above chance (37.5% / 36.2%; chance 25%; PPA-T 22.9% / 25.0%). In a new polarity study, raters chose the pole named by the explanation 68.4% / 69.2% of the time (chance 50%), and within a concept, their choices switched with the model's sign (+34 points, p < .001). (App. L)

(3) Images the model gets wrong (w8Cr Q3; LQ6R). In a new human study without the error filter and with predicted bands, on mispredicted images Task 2 accuracy stays above chance (43.3% / 41.7%), and FaTeX keeps its advantage in all but one comparison (Task 1 vs. PPA-T on PARA, 46.7%). (App. K)

(4) Accuracy cost (Si64 Q1, W2). The cost comes from the additive read-out itself. Removing the gate, tag term or content control changes SRCC by at most .006, within the seed range, and fully fine-tuning four blocks does not help; these components change the explanations instead. Against PPA-T, the closest decomposable model, FaTeX gives up .007–.026 SRCC on five datasets, but humans prefer its explanations on both datasets, as does the LLM judge on the other four. The Discussion now states this trade-off. (App. F, G; Fig. S6)

(5) Robustness (Si64 Q2; o1fi W4). Regenerating the pool with three Gemini seeds changes the wording (13% / 15% identical pairs) but not accuracy (SD .006 / .002). Contributions are stable under perturbed centering and gains (rank correlation ≥ .95) and after retraining (.84 / .88, vs. .50 / .65 between images). (App. G, M)

(6) Coverage (LQ6R; o1fi Minor-W1). FaTeX transfers to PAMELA and AADB with 95–97% of the target-trained SRCC, a smaller drop than PPA-T or MLP; no method transfers to APDDv2 or TAD66K, whose ratings are defined differently (App. O). Empty cells in Table 1 are explained per method (App. D, H).

3. Remaining limitations (stated in the revision)

  • Polarity is measured directly only on images with small prediction errors; for mispredicted images, the evidence is indirect, a scope o1fi accepted.
  • On mispredicted PARA images, FaTeX's preference over PPA-T is no longer clear.
  • The top concepts partly change with the seed, and the pool's wording is not unique.

Thanks to the reviewers' careful and constructive comments, the scope of our claims is now stated precisely, and the contribution holds within it: an IAA model whose prediction is an exact sum of named concepts, with explanations that humans prefer, that single out their image, and whose named poles agree with human judgment. Thank you for your time and consideration.

Best regards, The Authors

References

  • Espinosa Zarlenga. In Defense of Information Leakage in Concept-based Models. ICML 2026.
  • Jacovi and Goldberg. Towards Faithfully Interpretable NLP Systems: How Should We Define and Evaluate Faithfulness? ACL 2020.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.