- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of current MLLMs in reasoning about visual persuasion. To address these challenges, the authors propose a dual-axis rationale supervision approach based on evidence polarity and visual granularity, and evaluate its effectiveness against various post-training strategies in a controlled setting. Furthermore, the paper introduces three complementary metrics for assessing rationale faithfulness for visual persuasion, demonstrating that higher prediction performance does not necessarily correspond to more faithful reasoning, and highlighting the importance of evaluating reasoning effectiveness and faithfulness separately.
Critical strengths
- The proposed dual-axis rationale supervision achieves competitive and generally improved classification performance across MLLM families, while mitigating the tendency to overpredict the persuasive class.
- The paper goes beyond classification accuracy by evaluating rationale faithfulness along 3 complementary metrics. In particular, the use of counterfactual image editing to assess decision sensitivity, supported by human validation and control experiments, is an interesting approach. The observed discrepancy between prediction performance and faithfulness also provides useful insights.
Critical weaknesses
-
The teacher-generated rationales are conditioned on ground-truth labels, raising concerns that teachers may produce plausible post-hoc justifications rather than genuinely grounded reasoning. Although the paper provides some rationale-quality analyses and single-prompt ablations, it remains unclear whether the performance gains stem from diverse reasoning perspectives or simply increased teacher-generated supervision. To better isolate the contribution of dual-axis supervision, I suggest comparing single- and multi-perspective rationales under matched training budgets (e.g., data scale), as well as evaluating whether training on human-validated, high-quality rationales yields different results.
-
The proposed faithfulness evaluation framework appears somewhat heuristic, and its specific contribution to visual persuasion reasoning remains unclear. In particular, the groundedness metric primarily verifies concrete visual claims without sufficiently examining how affective, symbolic, or compositional cues support persuasive reasoning. This raises questions about how the proposed evaluation differs from existing general-purpose visual grounding and faithfulness metrics. A more comprehensive human evaluation across different types of persuasive visual cues would help establish the domain-specific validity and reliability of the framework.
Questions for authors
Q1. Message-Relevance Shortcut. The proposed message-relevance shortcut is mainly supported by qualitative examples. Could the authors provide quantitative evidence of how consistently this phenomenon occurs across current models and whether Reasoning-SFT effectively mitigates it?
Q2. Teacher Rationale Quality. Given that teacher rationales are generated conditioned on ground-truth labels, how do the authors ensure that these rationales reflect valid visual evidence rather than plausible post-hoc justifications?
Q3. Classification Improvement vs. Faithfulness. Could the authors further explain why Reasoning-SFT improves classification performance (e.g., why mitigate over-persuasive prediction), even though higher accuracy does not necessarily lead to more faithful reasoning?
Other comment: I believe this paper makes a valuable contribution to advancing research on visual persuasion. I hope the authors will adequately address my main concerns during the rebuttal process, which would strengthen my confidence in recommending acceptance.
10 comments
W1. Rationale Quality Control and Dual-Axis Supervision under Matched Training Budgets (1/2)
1. Post-hoc Justification and Rationale Quality
We acknowledge the reviewer's concern that conditioning teacher-generated rationales on ground-truth labels may introduce post-hoc justifications. Although this possibility cannot be completely ruled out, we conducted additional experiments to investigate how rationale quality affects downstream classification performance.
Specifically, we constructed a quality-controlled rationale dataset through the following procedure:
- a) Initial rationale quality assessment: We evaluated teacher-generated rationales using our improved groundedness judge (see our response to Reviewer wPkk, W2-1) and consistency metric. We found that 22.6% of rationales failed at least one criterion and selected them for further assessment and potential revision (b–d), retaining the remaining 77.4% unchanged. Although our sensitivity metric could be effective for rationale quality assessment, applying it to the entire training dataset would require substantial computational resources. Given that editing a single image takes approximately 90 seconds on an A100 GPU, evaluating 1,520 samples × 4 rationale categories × 4 backbones × 5 seeds would require approximately 121,600 image edits (3,040 A100 GPU-hours). Given this computational cost, we did not use the sensitivity metric for this filtering.
- b) Human evaluation: To validate our LLM-based rationale-quality scorer, we first prepared human annotations. Specifically, three independent annotators evaluated rationale quality on 240 image–rationale pairs randomly sampled from our training datasets, with equal representation across teacher models and reasoning perspectives defined by the polarity and granularity axes. Each rationale was rated on a 1–5 scale, considering the image, target message, model prediction, and rationale.
- c) Validation of G-Eval: We compared human ratings with GPT-5-based G-Eval (Liu et al., 2023) scores. Among samples receiving an average G-Eval score ≥4, 98.5% also received an average human rating ≥4, demonstrating strong alignment in identifying high-quality rationales.
- d) Rationale revision and human verification: Rationales scoring below 4 were iteratively revised by GPT-5 using judge feedback to correct unsupported visual claims and reasoning errors while preserving the original judgment and reasoning perspective. We allowed up to five revision iterations until the score reached ≥4, with an average of 1.03 iterations and a maximum of 3 observed in practice. All revised rationales subsequently underwent human verification.
Table 1. Balanced Accuracy (%) averaged over five seeds using dual-axis supervision.
| Stdent | Data | Dual-axis |
|---|---|---|
| Qwen | Original | 75.4 |
| Quality-controlled | 74.1 | |
| Phi | Original | 70.9 |
| Quality-controlled | 71.7 | |
| Gemma | Original | 72.6 |
| Quality-controlled | 73.1 | |
| LLaVA | Original | 68.6 |
| Quality-controlled | 67.0 |
Improving rationale quality did not consistently translate into better persuasiveness classification performance. Under matched training budgets, quality-controlled dual-axis supervision improved balanced accuracy for Phi (70.9 → 71.7) and Gemma (72.6 → 73.1), whereas performance decreased for Qwen (75.4 → 74.1) and LLaVA (68.6 → 67.0), as shown in table above.
To better understand these mixed results, we analyzed the types of errors in the original teacher-generated rationales. Among these correction reasons, unsupported inferences and logical leaps accounted for a slightly higher proportion in Qwen/LLaVA than in Phi/Gemma (39.7% vs. 37.6%), whereas hallucinations were more prevalent in Phi/Gemma (21.9% vs. 17.7%).
One possible explanation is that correcting hallucinated visual details in Phi/Gemma helped replace inaccurate claims with concrete visual evidence. In contrast, correcting unsupported inferences and logical leaps in Qwen/LLaVA sometimes introduced visual details that could support the target message, even when the final judgment remained negative. For instance, a rationale noting missing window-cleaning cues was revised to describe dirty windows and cleaning tools, while still concluding that the image failed to motivate cleaning. Such revisions may have improved visual grounding but weakened support for negative predictions, potentially introducing mixed supervision signals. This interpretation is consistent with the decrease in recall for the "No" class following quality control, although further analysis is needed to establish a causal relationship.
Overall, these findings suggest that improvements in rationale quality do not necessarily yield corresponding gains in classification accuracy, and that the original teacher-generated rationales may retain useful predictive information.
We have also added detailed experimental settings and further analyses of the results to Appendix E.4 to clarify these findings.
W1. Rationale Quality Control and Dual-Axis Supervision under Matched Training Budgets (2/2)
2. Dual-Axis vs. Single-Axis Supervision under Matched Training Budgets
To distinguish whether the improvements arise from diverse reasoning perspectives or simply increased teacher-generated supervision, we conducted additional experiments comparing single-axis and dual-axis supervision under matched training token budgets. Specifically, we compared five settings: polarity-only supervision with either global or* local* granularity fixed, granularity-only supervision with either including or excluding polarity fixed, and dual-axis supervision combining both dimensions. Each single-axis setting uses two distinct prompt formats.
To ensure a fair comparison, all settings used 1,520 image–rationale pairs. Each single-axis setting contained 1,520 samples drawn from two relevant rationale categories (760 each), whereas dual-axis supervision contained 1,520 samples drawn equally from all four categories (380 each). We further controlled the training token budget by (1) calculating the average training sequence length using each student backbone's tokenizer, (2) keeping batch size and gradient accumulation fixed, and (3) adjusting the number of training steps for each setting so that the total number of tokens processed during training is equal across all settings.
Table 2. Balanced Accuracy (%) averaged over five seeds under matched training budgets.
| Backbone | Data | P: Global | P: Local | G: Incl. | G: Excl. | Dual-axis | Δ |
|---|---|---|---|---|---|---|---|
| Qwen | Original | 73.7 | 72.2 | 73.6 | 73.3 | 75.4 | +1.7 |
| Quality-controlled | 72.5 | 72.9 | 73.4 | 71.8 | 74.1 | +0.7 | |
| Phi | Original | 66.4 | 61.8 | 63.9 | 69.4 | 70.9 | +1.5 |
| Quality-controlled | 71.4 | 69.5 | 68.9 | 71.2 | 71.7 | +0.3 | |
| Gemma | Original | 70.2 | 71.6 | 69.7 | 71.0 | 72.6 | +1.0 |
| Quality-controlled | 70.3 | 72.4 | 68.6 | 68.6 | 73.1 | +0.7 | |
| LLaVA | Original | 68.2 | 64.0 | 66.0 | 66.1 | 68.6 | +0.4 |
| Quality-controlled | 68.3 | 64.6 | 68.3 | 67.9 | 67.0 | -1.3 |
P: Polarity-only; G: Granularity-only; Incl./Excl.: Including/Excluding. Bold indicates the highest performance within each backbone and data condition. Δ denotes Dual-axis minus the best Single-axis setting (percentage points).
a) Results on original rationales
Dual-axis supervision achieved the highest average balanced accuracy across all four backbones. Compared with the best single-axis setting, it improved performance by +1.7 percentage points on Qwen, +1.5 on Phi, +1.0 on Gemma, and +0.4 on LLaVA. These findings demonstrate that dual-axis supervision achieves higher classification performance than single-axis supervision, even when training data scale and token budgets are controlled.
b) Results on quality-controlled rationales
We further evaluated whether dual-axis supervision is effective after rationale quality control. Dual-axis supervision continued to outperform the best single-axis setting on Qwen (+0.7 percentage points), Phi(+0.3), and Gemma (+0.7), whereas LLaVA showed a decrease (-1.3). These results suggest that dual-axis supervision may remain effective after rationale quality control, although its impact differs across backbones.
We appreciate the reviewer's insightful comment, which motivated this additional analysis. We have added a summary of the experimental results to the Section 3.4.2, along with detailed experimental settings, including the human evaluation procedure, and further analysis of the results in Appendix E.4.
Review Summary
We sincerely appreciate your constructive review and your kind remark that our paper makes a valuable contribution to research on visual persuasion. We are pleased that you find our rationale supervision effective, with competitive and generally improved performance across MLLM families. We also thank you for highlighting our human-validated counterfactual image editing, as well as the useful insights from the observed gap between prediction performance and faithfulness.