- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies interpretable image aesthetic assessment (IAA), where standard concept bottleneck assumptions are problematic because aesthetic concepts are evaluative, content-dependent, and lack ground-truth concept annotations. It proposes FaTeX, which constructs dataset-specific pools of antonymic textual concepts using an LLM, filters them by rating association after controlling for image content, and uses orthogonal matching pursuit to reduce redundancy. A frozen CLIP encoder supplies signed concept responses and an applicability gate, while a trainable image encoder allocates per-image concept weights and a magnitude. The prediction is an exact sum of named concept contributions. Experiments on six datasets show that FaTeX remains within 0.035 SRCC of the best predictive methods on average and improves over C2J, although PPA-T is more accurate on five of six datasets. Human and LLM studies on LAPIS and PARA generally prefer FaTeX's top-three concept summaries and find them more image-specific than PPA-T; C2J remains tied on LAPIS, where it uses expert style annotations. My main concern is that exact additivity establishes an arithmetic decomposition, but not necessarily semantic faithfulness: the learned weights and magnitude can carry unnamed task information, and the displayed top-three text covers only about one third of the total absolute contribution.
Critical strengths
-
The problem formulation and research story are excellent. The paper clearly explains why aesthetic concepts lack the localization, objective labels, and content invariance assumed by standard CBMs. Content-controlled selection, a hierarchy of collection-wide and content-specific concepts, and per-image applicability gating are well motivated responses to these failures. This makes the work scientifically meaningful even though many individual building blocks have precedents.
-
The empirical study is broad, transparent, and unusually thoughtful. It covers six datasets, two backbones, predictive and concept-based baselines, human and LLM evaluation, image-identification tests, and extensive ablations. Task 2 controls the distractors' content tag and rating band, making it a useful test of specificity. The authors also report unfavorable results, including that FaTeX is never the best predictive model, and provide detailed protocols and limitations that materially improve assessability and reproducibility.
Critical weaknesses
-
Exact additivity is insufficient to establish semantic faithfulness. The learned allocation p(t|x) and magnitude rho(x) provide unrestricted image-dependent channels around the frozen concept response. The paper itself observes that borrowing another image's allocation lowers SRCC by roughly 0.05/0.10 and acknowledges that this may be unnamed task information. Without a dedicated concept/label leakage test or semantic interventions, large values of c_t show that a term is numerically important, not that the term's natural-language name captures the actual reason for the prediction. The deletion experiment inherits this limitation. Moreover, the evaluated top-three text covers only 31.5%/34.0% of absolute contribution and shows no values, so “the explanation is the prediction” is true only for the complete internal vector, not for the textual explanation presented to users.
-
The human evidence is strong as a relative vocabulary comparison but limited as an end-to-end explanation evaluation. The study first restricts evaluation to test images on which every compared model predicts within half a test-set standard deviation of the target, yet does not report the eligible fraction. This can exclude difficult or failure cases. The explanation template uses the ground-truth low/average/high band rather than the model prediction, and only LAPIS and PARA are evaluated. Each comparison has two raters per image, uses forced choice without a neutral option, and the many pair-by-question intervals receive no multiplicity correction. The core ablations are also retrained only once. These choices do not invalidate the relative result, but they make the paper's general claims about generated explanations and statistical significance too broad.
Questions for authors
-
Can the authors provide a dedicated information-leakage analysis that separates what can be predicted from d(x,t), p(t|x), and rho(x), and tests whether weights retain the target after concept responses or concept identities are randomized? Concept-level counterfactual interventions or a variant whose weights depend only on verified concept values would materially affect my view of the faithfulness claim.
-
Since the displayed top-three concepts account for only 31.5%/34.0% of total absolute contribution and show no numerical values, what completeness or fidelity guarantee does the actual textual explanation provide? Please report its residual contribution or reconstruction error and compare different explanation lengths.
-
What number and fraction of each test set pass the all-model error filter used for the human study? Do the main human results persist on an unfiltered random test sample when the template uses the model's predicted band rather than the ground-truth band? This would determine how far the reported explanation quality generalizes to deployment and model failures.
6 comments
Response to Reviewer w8Cr (Part 2/5)
- What number and fraction of each test set pass the all-model error filter used for the human study? Do the main human results persist on an unfiltered random test sample when the template uses the model's predicted band rather than the ground-truth band? This would determine how far the reported explanation quality generalizes to deployment and model failures.
[Response] In short, 35.9% (LAPIS) and 52.6% (PARA) of test images passed the original filter. In a new human evaluation without the filter and with the predicted band, raters identified the image from FaTeX's explanations (Task 2) at least as often as in the original setting, and above chance even on images the model got wrong. Preference (Task 1) held on LAPIS, where FaTeX was chosen about as often on images the model got wrong as on the others, but narrowed on the PARA images the model got wrong, where the difference from PPA-T was no longer clear.
Images for the human study were drawn from those on which every compared model predicts within half a test-set SD of the target. The numbers and fractions that pass are:
| Dataset | Test | Pass (all models) | Pass rate by band (low / average / high) |
|---|---|---|---|
| LAPIS | 2,345 | 843 (35.9%) | 32.0% / 40.8% / 35.0% |
| PARA | 3,000 | 1,577 (52.6%) | 43.2% / 69.4% / 45.4% |
The filter keeps differences in predictive accuracy out of the comparison of explanations. As the reviewer notes, however, it limits our conclusions to images on which the models are roughly correct.
The original study cannot answer this question. All of its images are ones on which every model predicted correctly, so it contains no image on which the model is wrong. Also, the template's level word was the ground-truth band rather than the model's prediction. This kept the level word identical across the two explanations being compared, so that raters compared only the concepts; with predicted bands, the level word would change whenever the methods' predictions differ. We therefore ran an additional human study to measure the two things you ask about:
- (a) Do the original conclusions hold as the explanations would be used (no filter, level word from the model's predicted band)?
- (b) Are the explanations still useful on images the model gets wrong?
Tasks, questions, checks and exclusion criteria are those of the original study; only two things changed:
- Images: without the filter, we drew 60 images at random from those FaTeX misses (|error| > 0.5 SD) and 60 from those it does not. A simple random sample of 120 images would contain missed images only at their share of the test set (16.3% on PARA, 35.7% on LAPIS), about 20 images on PARA. To examine model failures, which the question also raises, at the same time, we oversampled missed images so that (b) is answered with enough images. Weighting the two strata by their shares of the test set estimates the same quantities as an unfiltered random sample, and we answer (a) with this estimate (the "unfiltered (weighted estimate)" columns of the table).
- Level word: each model's predicted band instead of the ground-truth band.
As this is a follow-up study, we reduced the raters per item from two to one, which widens the confidence intervals.
| LAPIS: original | LAPIS: unfiltered (weighted estimate) | LAPIS: model wrong | PARA: original | PARA: unfiltered (weighted estimate) | PARA: model wrong | |
|---|---|---|---|---|---|---|
| Task 1: FaTeX preferred (Q0, vs w/o ) | 63.7% | 63.5% | 66.7% | 65.4% | 70.3% | 55.0% |
| Task 1: same (vs PPA-T) | 71.2% | 77.9% | 80.0% | 65.0% | 56.4% | 46.7% |
| Task 1: same (vs C2J) | 56.2% | 58.9% | 60.0% | 69.6% | 72.5% | 68.3% |
| Task 2: FaTeX accuracy (chance 25%) | 37.5% | 45.5% | 43.3% | 36.2% | 37.5% | 41.7% |
"Model wrong" is the stratum of 60 images on which FaTeX misses by more than 0.5 SD; "original" is the submitted human study (two raters per item).
(a) Unfiltered test images. In the weighted estimates, Task 2 accuracy is at or above the original setting. On LAPIS, the 95% CIs of the differences from all three methods exclude zero; on PARA, the differences from C2J and PPA-T have the same sign as before, with CIs that just include zero. In Task 1, LAPIS is close to the original setting. On PARA, FaTeX is preferred at or above the original rate over w/o and C2J, but its preference over PPA-T falls to 56.4% from 65.0%, with a CI that includes 50%.
(continued in the next comment)
Response to Reviewer w8Cr (Part 3/5)
(continued from the previous comment)
(b) Images on which the model is wrong. In Task 2, on the 60 images the model misses, raters identified the correct image from FaTeX's explanation 43.3% of the time on LAPIS (95% CI 31.7–56.7%) and 41.7% on PARA (30.0–55.0%), above chance (25%) and no lower than on the 60 images it predicts well. In Task 1, LAPIS raters preferred FaTeX about as often on missed images as on the others, whereas on PARA FaTeX's advantage narrowed on missed images, to 46.7% against PPA-T (58.3% on well-predicted images).
With one rater per item, the confidence intervals are wider than in the original setting, and more comparisons have intervals that include chance or zero. Even so, FaTeX keeps the same direction of advantage as in the original setting in almost every comparison. The exception is Task 1 against PPA-T on PARA, where the preference reverses on images the model gets wrong (46.7%).
[Action] We will add the pass-rate table to the appendix. We will also add the unfiltered, predicted-band human evaluation to the appendix, and state in the human-evaluation section of the main text how far the results extend.
Response to Reviewer w8Cr (Part 1/5)
We thank the reviewer for proposing, with deep expertise in interpretable AI, concrete ways to test our claims, such as leakage tests and counterfactual interventions. Thanks to these comments, the manuscript now states the faithfulness claim within a more accurate scope. We are also grateful that the reviewer recognized the integrity of this work, including our disclosure of unfavorable results. We respond to each point below. For Q1, additional experiments are still in progress; we answer the other points first and will post our response to Q1 as soon as they are complete.
[Response] We agree. The exact decomposition holds for the full vector of concept contributions, and the three-concept text is a summary of it.
In short, the displayed three-concept text preserves the direction of the prediction (in 92% of images) and its ranking (SRCC .94 / .92 with the full prediction), but not its magnitude: about 60% of the deviation from the baseline is carried by concepts that are not named. k = 3 is a choice in favour of readability.
To quantify what the text guarantees, we measured, for each explanation length k, how well the sum Sk of the k largest contributions reproduces the prediction y^=a+∑tct, where a is the baseline. We used the runs explained in the human study, over the full test sets (LAPIS 2,345, PARA 3,000), without retraining. Values are LAPIS / PARA.
The residual falls gradually with k (still about 30% at k = 10), with no clear elbow. The 120 human-study images show the same pattern (relative residual at k = 3: .56 / .61; sign agreement 92% / 91%). The text therefore guarantees the direction and ranking of the prediction, not its magnitude. Where the magnitude is needed, the full contribution vector, for which the exact decomposition holds, can be shown with its values.
Restricting the text to a few concept names is to some extent unavoidable for an explanation that people read and judge. Label-Free CBM (Oikarinen et al., 2023) likewise presents only a subset of the contributing concepts, by name, in its human evaluation. Our k = 3 follows the same reasoning: the text is a readable summary of the full contribution vector.
[Action] We will add the table (k = 1, 3, 5, 10, all) to the appendix. We will revise the annotation "the explanation is the prediction" in Fig. 2 and the claim in the text to state that the exact decomposition holds for the full contribution vector, and that the displayed three-concept text is a summary that preserves the direction and ranking of the prediction but not most of its magnitude.