acceptodds
Under review as a conference paper at ICLR 2027

Can MLLMs Reason About Visual Persuasion? Evaluating the Efficacy and Faithfulness of Reasoning

Abstract

Persuasive visuals play a central role in advertising, public communication, and online media, making it increasingly important to understand whether an image is persuasive and why. However, current Multimodal Large Language Models (MLLMs) have limited ability to reason about visual persuasion. Our analysis reveals that models often rely on a shortcut—selectively citing easily recognizable visual elements and treating their mere presence as evidence of persuasiveness—rather than reasoning over the diverse visual cues in an image to determine how and why they contribute to persuasiveness. To address this limitation, we propose (1) a fine-tuning approach that trains models on rationales reflecting diverse perspectives on visual persuasiveness, and (2) an evaluation framework that measures the faithfulness of models' rationales through three complementary metrics. Fine-tuning on multi-perspective rationales improves persuasiveness prediction and reasoning effectiveness. However, our evaluation framework reveals a discrepancy between prediction performance and rationale faithfulness, showing that higher prediction performance does not necessarily correspond to more faithful reasoning. These findings highlight the need to evaluate effectiveness and faithfulness separately and inform future directions for improving both the training and evaluation of visual persuasion reasoning.

Primary area:
datasets and benchmarks
Keywords:
Visual Persuasion, Multimodal Reasoning, Dataset Construction, Evaluation Framework, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KFDAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the limitations of current MLLMs in reasoning about visual persuasion. To address these challenges, the authors propose a dual-axis rationale supervision approach based on evidence polarity and visual granularity, and evaluate its effectiveness against various post-training strategies in a controlled setting. Furthermore, the paper introduces three complementary metrics for assessing rationale faithfulness for visual persuasion, demonstrating that higher prediction performance does not necessarily correspond to more faithful reasoning, and highlighting the importance of evaluating reasoning effectiveness and faithfulness separately.

Critical strengths
  1. The proposed dual-axis rationale supervision achieves competitive and generally improved classification performance across MLLM families, while mitigating the tendency to overpredict the persuasive class.
  2. The paper goes beyond classification accuracy by evaluating rationale faithfulness along 3 complementary metrics. In particular, the use of counterfactual image editing to assess decision sensitivity, supported by human validation and control experiments, is an interesting approach. The observed discrepancy between prediction performance and faithfulness also provides useful insights.
Critical weaknesses
  1. The teacher-generated rationales are conditioned on ground-truth labels, raising concerns that teachers may produce plausible post-hoc justifications rather than genuinely grounded reasoning. Although the paper provides some rationale-quality analyses and single-prompt ablations, it remains unclear whether the performance gains stem from diverse reasoning perspectives or simply increased teacher-generated supervision. To better isolate the contribution of dual-axis supervision, I suggest comparing single- and multi-perspective rationales under matched training budgets (e.g., data scale), as well as evaluating whether training on human-validated, high-quality rationales yields different results.

  2. The proposed faithfulness evaluation framework appears somewhat heuristic, and its specific contribution to visual persuasion reasoning remains unclear. In particular, the groundedness metric primarily verifies concrete visual claims without sufficiently examining how affective, symbolic, or compositional cues support persuasive reasoning. This raises questions about how the proposed evaluation differs from existing general-purpose visual grounding and faithfulness metrics. A more comprehensive human evaluation across different types of persuasive visual cues would help establish the domain-specific validity and reliability of the framework.

Questions for authors

Q1. Message-Relevance Shortcut. The proposed message-relevance shortcut is mainly supported by qualitative examples. Could the authors provide quantitative evidence of how consistently this phenomenon occurs across current models and whether Reasoning-SFT effectively mitigates it?

Q2. Teacher Rationale Quality. Given that teacher rationales are generated conditioned on ground-truth labels, how do the authors ensure that these rationales reflect valid visual evidence rather than plausible post-hoc justifications?

Q3. Classification Improvement vs. Faithfulness. Could the authors further explain why Reasoning-SFT improves classification performance (e.g., why mitigate over-persuasive prediction), even though higher accuracy does not necessarily lead to more faithful reasoning?


Other comment: I believe this paper makes a valuable contribution to advancing research on visual persuasion. I hope the authors will adequately address my main concerns during the rebuttal process, which would strengthen my confidence in recommending acceptance.

10 comments
AuthorsOpenReview ↗

Review Summary

We sincerely appreciate your constructive review and your kind remark that our paper makes a valuable contribution to research on visual persuasion. We are pleased that you find our rationale supervision effective, with competitive and generally improved performance across MLLM families. We also thank you for highlighting our human-validated counterfactual image editing, as well as the useful insights from the observed gap between prediction performance and faithfulness.

AuthorsOpenReview ↗

W1. Rationale Quality Control and Dual-Axis Supervision under Matched Training Budgets (1/2)

1. Post-hoc Justification and Rationale Quality

We acknowledge the reviewer's concern that conditioning teacher-generated rationales on ground-truth labels may introduce post-hoc justifications. Although this possibility cannot be completely ruled out, we conducted additional experiments to investigate how rationale quality affects downstream classification performance.

Specifically, we constructed a quality-controlled rationale dataset through the following procedure:

  • a) Initial rationale quality assessment: We evaluated teacher-generated rationales using our improved groundedness judge (see our response to Reviewer wPkk, W2-1) and consistency metric. We found that 22.6% of rationales failed at least one criterion and selected them for further assessment and potential revision (b–d), retaining the remaining 77.4% unchanged. Although our sensitivity metric could be effective for rationale quality assessment, applying it to the entire training dataset would require substantial computational resources. Given that editing a single image takes approximately 90 seconds on an A100 GPU, evaluating 1,520 samples × 4 rationale categories × 4 backbones × 5 seeds would require approximately 121,600 image edits (3,040 A100 GPU-hours). Given this computational cost, we did not use the sensitivity metric for this filtering.
  • b) Human evaluation: To validate our LLM-based rationale-quality scorer, we first prepared human annotations. Specifically, three independent annotators evaluated rationale quality on 240 image–rationale pairs randomly sampled from our training datasets, with equal representation across teacher models and reasoning perspectives defined by the polarity and granularity axes. Each rationale was rated on a 1–5 scale, considering the image, target message, model prediction, and rationale.
  • c) Validation of G-Eval: We compared human ratings with GPT-5-based G-Eval (Liu et al., 2023) scores. Among samples receiving an average G-Eval score ≥4, 98.5% also received an average human rating ≥4, demonstrating strong alignment in identifying high-quality rationales.
  • d) Rationale revision and human verification: Rationales scoring below 4 were iteratively revised by GPT-5 using judge feedback to correct unsupported visual claims and reasoning errors while preserving the original judgment and reasoning perspective. We allowed up to five revision iterations until the score reached ≥4, with an average of 1.03 iterations and a maximum of 3 observed in practice. All revised rationales subsequently underwent human verification.

Table 1. Balanced Accuracy (%) averaged over five seeds using dual-axis supervision.

StdentDataDual-axis
QwenOriginal75.4
Quality-controlled74.1
PhiOriginal70.9
Quality-controlled71.7
GemmaOriginal72.6
Quality-controlled73.1
LLaVAOriginal68.6
Quality-controlled67.0

Improving rationale quality did not consistently translate into better persuasiveness classification performance. Under matched training budgets, quality-controlled dual-axis supervision improved balanced accuracy for Phi (70.9 → 71.7) and Gemma (72.6 → 73.1), whereas performance decreased for Qwen (75.4 → 74.1) and LLaVA (68.6 → 67.0), as shown in table above.

To better understand these mixed results, we analyzed the types of errors in the original teacher-generated rationales. Among these correction reasons, unsupported inferences and logical leaps accounted for a slightly higher proportion in Qwen/LLaVA than in Phi/Gemma (39.7% vs. 37.6%), whereas hallucinations were more prevalent in Phi/Gemma (21.9% vs. 17.7%).

One possible explanation is that correcting hallucinated visual details in Phi/Gemma helped replace inaccurate claims with concrete visual evidence. In contrast, correcting unsupported inferences and logical leaps in Qwen/LLaVA sometimes introduced visual details that could support the target message, even when the final judgment remained negative. For instance, a rationale noting missing window-cleaning cues was revised to describe dirty windows and cleaning tools, while still concluding that the image failed to motivate cleaning. Such revisions may have improved visual grounding but weakened support for negative predictions, potentially introducing mixed supervision signals. This interpretation is consistent with the decrease in recall for the "No" class following quality control, although further analysis is needed to establish a causal relationship.

Overall, these findings suggest that improvements in rationale quality do not necessarily yield corresponding gains in classification accuracy, and that the original teacher-generated rationales may retain useful predictive information.

We have also added detailed experimental settings and further analyses of the results to Appendix E.4 to clarify these findings.

AuthorsOpenReview ↗

W1. Rationale Quality Control and Dual-Axis Supervision under Matched Training Budgets (2/2)

2. Dual-Axis vs. Single-Axis Supervision under Matched Training Budgets

To distinguish whether the improvements arise from diverse reasoning perspectives or simply increased teacher-generated supervision, we conducted additional experiments comparing single-axis and dual-axis supervision under matched training token budgets. Specifically, we compared five settings: polarity-only supervision with either global or* local* granularity fixed, granularity-only supervision with either including or excluding polarity fixed, and dual-axis supervision combining both dimensions. Each single-axis setting uses two distinct prompt formats.

To ensure a fair comparison, all settings used 1,520 image–rationale pairs. Each single-axis setting contained 1,520 samples drawn from two relevant rationale categories (760 each), whereas dual-axis supervision contained 1,520 samples drawn equally from all four categories (380 each). We further controlled the training token budget by (1) calculating the average training sequence length using each student backbone's tokenizer, (2) keeping batch size and gradient accumulation fixed, and (3) adjusting the number of training steps for each setting so that the total number of tokens processed during training is equal across all settings.

Table 2. Balanced Accuracy (%) averaged over five seeds under matched training budgets.

BackboneDataP: GlobalP: LocalG: Incl.G: Excl.Dual-axisΔ
QwenOriginal73.772.273.673.375.4+1.7
Quality-controlled72.572.973.471.874.1+0.7
PhiOriginal66.461.863.969.470.9+1.5
Quality-controlled71.469.568.971.271.7+0.3
GemmaOriginal70.271.669.771.072.6+1.0
Quality-controlled70.372.468.668.673.1+0.7
LLaVAOriginal68.264.066.066.168.6+0.4
Quality-controlled68.364.668.367.967.0-1.3

P: Polarity-only; G: Granularity-only; Incl./Excl.: Including/Excluding. Bold indicates the highest performance within each backbone and data condition. Δ denotes Dual-axis minus the best Single-axis setting (percentage points).

a) Results on original rationales

Dual-axis supervision achieved the highest average balanced accuracy across all four backbones. Compared with the best single-axis setting, it improved performance by +1.7 percentage points on Qwen, +1.5 on Phi, +1.0 on Gemma, and +0.4 on LLaVA. These findings demonstrate that dual-axis supervision achieves higher classification performance than single-axis supervision, even when training data scale and token budgets are controlled.

b) Results on quality-controlled rationales

We further evaluated whether dual-axis supervision is effective after rationale quality control. Dual-axis supervision continued to outperform the best single-axis setting on Qwen (+0.7 percentage points), Phi(+0.3), and Gemma (+0.7), whereas LLaVA showed a decrease (-1.3). These results suggest that dual-axis supervision may remain effective after rationale quality control, although its impact differs across backbones.

We appreciate the reviewer's insightful comment, which motivated this additional analysis. We have added a summary of the experimental results to the Section 3.4.2, along with detailed experimental settings, including the human evaluation procedure, and further analysis of the results in Appendix E.4.

Reviewer_WFfTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates MLLMs on visual persuasion reasoning. It identifies a "message-relevance shortcut" in current models. Contributions include a dual-axis rationale fine-tuning method and a 3-metric faithfulness evaluation framework (groundedness, consistency, sensitivity). Results show fine-tuning improves accuracy, groundedness, and consistency, but degrades sensitivity.

Critical strengths
  1. The 3-axis faithfulness evaluation framework, especially the image-editing pipeline, is an innovative way to audit MLLM reasoning.
  2. The experimental design is relatively rigorous and highly reliable. The authors have made detailed arrangements in dataset construction, model comparison, metric validation, and controlled experiments to minimize bias caused by subjectivity and chance.
Critical weaknesses
  1. The dual-axis design's unique causal contribution is not rigorously established, and simpler alternatives are not ruled out. The paper lacks a single-axis model trained with a matched number of prompt formats and token budget as a critical control.
  2. The faithfulness evaluation framework has highly uneven measurement quality across its three dimensions and lacks the ability to distinguish among multiple valid reasoning paths. Groundedness judged by GPT-5 reaches only κ = 0.145 against human votes, consistency shows a ceiling effect above 90%, and sensitivity retains a control-level response of ~0.48 under random edits. Moreover, the framework evaluates faithfulness relative to the model's own prediction and cannot distinguish whether a rationale follows a valid alternative reasoning path. These concerns about construct validity are independent of the data-quality and supervision gaps acknowledged in the future directions.
Questions for authors
  1. The coverage argument relies on an external embedding ψ(r), which is only a "gradient-free proxy", as acknowledged in Remark 1 (Appendix F.3). Given this gap, how to rule out that the gains of Reasoning-SFT simply come from a regularization effect of mixing multiple prompt formats rather than from the complementary nature of the two axes?
  2. Have the authors tested the models and faithfulness framework on the retained ambiguous samples (or on the full PVP dataset)? If the models' groundedness/consistency/sensitivity performance differs substantially on ambiguous samples, how would this affect the conclusions about the validity of the framework?
7 comments
AuthorsOpenReview ↗

Review Summary

We greatly appreciate your thoughtful and encouraging review. We are happy that you find our 3-axis faithfulness evaluation framework, especially the image-editing pipeline, an innovative way to audit MLLM reasoning. We are also grateful for your positive feedback on our experimental design, including the detailed arrangements in dataset construction, model comparison, metric validation, and controlled experiments.

AuthorsOpenReview ↗

W1. Contribution of Dual-Axis Supervision

To isolate the contribution of dual-axis supervision, we conducted additional ablation experiments across four student backbones and five random seeds under matched training budgets.

Experimental Setup

We compared five settings: polarity-only supervision with either global or* local* granularity fixed, granularity-only supervision with either including or excluding polarity fixed, and dual-axis supervision combining both dimensions. Each single-axis setting uses two distinct prompt formats.

To ensure a fair comparison, all settings used 1,520 image–rationale pairs. Each single-axis setting contained 1,520 samples drawn from two relevant rationale categories (760 each), whereas dual-axis supervision contained 1,520 samples drawn equally from all four categories (380 each). We further controlled the training token budget by (1) calculating the average training sequence length using each student backbone's tokenizer, (2) keeping batch size and gradient accumulation fixed, and (3) adjusting the number of training steps for each setting so that the total number of tokens processed during training is equal across all settings.

Results

The following table reports balanced accuracy (%) averaged over five seeds.

SupervisionPerspectiveQwenPhiGemmaLLaVA
Polarity-onlyGlobal73.766.470.268.2
Polarity-onlyLocal72.261.871.664.0
Granularity-onlyIncluding73.663.969.766.0
Granularity-onlyExcluding73.369.471.066.1
Dual-axisBoth75.470.972.668.6

Under these controlled conditions, dual-axis achieved the highest balanced accuracy across all four backbones, outperforming the strongest single-axis alternatives by 1.7, 1.5, 1.0, and 0.4 percentage points on Qwen, Phi, Gemma, and LLaVA, respectively. These results suggest that combining evidence polarity and visual granularity provides complementary benefits beyond simply increasing training data or token budgets.

We have added detailed experimental settings and results in Appendix E.4. We acknowledge the reviewer's concern and appreciate the feedback which helped us strengthen the empirical evaluation of dual-axis supervision.

AuthorsOpenReview ↗

W2. Reliability and Construct Validity of the Faithfulness Evaluation Framework

1. Groundedness Evaluation Reliability

To better understand the limitations of the original groundedness judge, we conducted an error analysis and found that it sometimes failed to distinguish visually supported claims from unsupported inferences. For example, it incorrectly accepted "security personnel" as grounded when the image only showed individuals in ordinary clothing with identification badges, without sufficient evidence of their roles.

Based on these findings, we refined the guidelines to explicitly distinguish observable visual attributes from inferred identities. We also added five few-shot examples to clarify ambiguous cases involving object states, contextual interpretations, and text in images. Finally, we replaced GPT-5 with GPT-6-astra and revalidated the judge, ensuring that the few-shot examples were excluded from the validation set.

As a result, cohen's κ increased from 0.145 to 0.418, with 80.1% balanced accuracy against human annotations. Re-evaluation across four backbones also preserved our main finding, with Reasoning-SFT achieving the highest average groundedness (79.3%).

We have updated the groundedness evaluation protocol in Section 4.1.1 and the corresponding results in Section 4.2.1, with further details provided in Appendix G.2.

2. Interpreting Consistency and Sensitivity

We consider high consistency scores (>90%) to be largely expected, as this metric evaluates whether rationales logically support the model's own predictions, rather than whether those predictions are correct. Label-consistent rationale supervision further contributes to this tendency. Thus, consistency primarily serves to identify logical contradictions that may occur occasionally.

For sensitivity, because it measures the direction of confidence changes, control scores near 50% are expected if unrelated edits are roughly equally likely to increase or decrease confidence in the original prediction. In contrast, rationale-guided edits achieved 66.7%, exceeding random (46.1%) and irrelevant edits (49.1%) by 20.6 and 17.6 percentage points, respectively (Appendix H.1, Table 20). These differences suggest that rationale-targeted edits have a stronger directional effect on predictions. The additional sensitivity analyses are summarized in Section 4.2.3, with detailed results in Appendix H. For groundedness, we have substantially improved its reliability and updated all numbers in the manuscript (see the first point in our response to W2).

3. Scope of Faithfulness Evaluation

We interpret this concern as asking two things: (1) whether the model’s own prediction is an appropriate reference for faithfulness evaluation, and (2) whether the framework can assess the validity or quality of different rationales that support the same prediction. Please let us know if we have misunderstood your point.

For (1), using the model’s own prediction is intentional. Our framework examines whether a rationale is grounded in the image, whether the rationale is consistent with the model’s prediction, and whether the rationale faithfully supports the model’s actual judgment. If consistency were instead evaluated against the ground-truth label, a rationale could receive a high score despite contradicting the model’s actual answer. This would not capture the relationship we aim to evaluate. We therefore assess prediction correctness separately and use the model’s own prediction as the reference for rationale–decision consistency and sensitivity. This choice aligns the evaluation with the intended construct of faithfulness.

For (2), the framework captures important aspects of rationale quality without assuming a single valid reasoning path. Different rationales supporting the same prediction are evaluated against their respective visual evidence, rather than a fixed reference explanation. Groundedness assesses not only the presence of cited objects but also the accuracy of their described attributes, states, and relationships. Therefore, it partly evaluates an important component of rationale quality. This allows the framework to assess different valid rationales based on their visual grounding, without assuming a single correct reasoning path.

Reviewer_wPkkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies reasoning capabilities of multimodal llm (MLLMs) on persuasive visual media. Visual persuasive is challenging as in contrast to natural images they require the model to reason beyond visual recognition and requires the model to interpret diverse visual cues to interpret persuasiveness. To this aim, the paper investigations are two fold: (1) they investigate sft and grpo on answer +- rational persuasive media; (2) it develops an evaluation framework to measure rational faithfullness through rationale-to-image grounding, rationale-to-decision consistency, and (3) rationale-to-decision sensitivity. For sensitivity the GPT-5 is first use to create image editing style prompts that is weakened and qwen-image-edit is utilized to generate the edited image the difference in the log-probability of the original prediction between the original and edited images.

Evlaatuion setup is comprehensive and includes several different multimodal LLMs and shows that rationale-based supervision improves prediction accuracy and rationale groundedness and consistency

Critical strengths
  • Interesting and underexplored problem. Studying whether MLLMs can reason about visual persuasion is important, particularly because persuasiveness depends on subjective interpretations and diverse visual cues and is more implicit compared to common visual understanding tasks, requirng the model to have strong complex reasoning capabilities,. T

  • Comprehensive evaluation attempt. The paper evaluates multiple MLLMs and training baselines and proposes three complementary faithfulness metrics. In particular, the use of counterfactual image editing to measure sensitivity to rationale-cited visual evidence is an interesting direction.

Critical weaknesses

**[W1] Limited novelty and insufficient evidence for the benefits of dual-axis rationale supervision.**The proposed training method largely combines existing rationale generation. Most importantly, Its improvements over standard SFT are relatively small (e.g,m 0.5, +1.1, etc) and even underperforms SFT on LLaVA. The standard deviation reported in supplement seems to be larger than performance imrovement, raising a conern on reliability of the findings -- I appreciate if authors could clarify and also include CI, p-value alogn with standard deviatoin to the main paper to better evaluate the effectiveness.

[W2] Reliability faithfulness evaluation framework.

  • I appreciate if authors could clarify groundedness judge performance & reliability: achieves only 57.1% balanced accuracy and Cohen's κ = 0.145 against human annotations (Table 12 in appendix), raising concerns about the reliability of the groundedness results.
  • The sensitivity metric uses Δ≥0\Delta \ge 0, regardless of its magnitude. Thus metric considers delta=0.001 and delta =0.1 as the same. Additionally, The sensitivity metric also depends on the image-editing model strength, hence if model cannot reliablty show the changes will increase the senstiivty score. Hence, I hthink some ablation on utilizing different (strong, weaker) editing models could be very informative
Questions for authors

In additional to weaknesses:

  1. how are annotators recruited? and what quality measure steps were taken into account?

  2. can authors provide some error analysis on *wrong rationale only * vs *wrong prediction * vs *wrong rationale and wrong prediction *? This could show how much each model can correclty reason and its reasoning impact the prediction and can support the papers claims and contributions

  3. While I appreciate the inclusion of several backbones in the experimental setup, could authors please clarify where the eval set focuses on AI-generated subset or the web-sourced images are also included? reporting some of the main results/analysis on both set saprately can further supporting the contributions and generalization of findings across both real and synthetic data.

  4. [Minor] Rationale supervision: Since Reasoning-SFT directly trains on label-consistent, teacher-generated rationales, improvements in groundedness and consistency may be expected. Could the authors clarify this point or discuss it in the limitation? also what are additional benefits that arises from dual-axis rationale diversity beyond ordinary rationale-supervised fine-tuning?

15 comments
AuthorsOpenReview ↗

Review Summary

Thank you for your careful and detailed review of our paper. We are glad that you find visual persuasion reasoning an important and underexplored problem, and that you consider our evaluation comprehensive, covering multiple MLLMs and training baselines. We also appreciate your recognition of our three complementary faithfulness metrics, in particular the counterfactual image-editing approach to measuring sensitivity, as an interesting direction.

AuthorsOpenReview ↗

W1. Limited Novelty and Statistical Reliability of Dual-Axis Supervision (1/2)

1. Contribution of Dual-Axis Rationale Supervision

Our main focus is to analyze and improve MLLMs' reasoning over visual persuasion, and thus our contribution lies primarily in designing rationale supervision for visual persuasion reasoning rather than proposing a new fine-tuning algorithm. Visual persuasion requires models to assess not only which visual elements are present, but also how they strengthen or weaken the intended message. However, existing MLLMs often rely on a message-relevance shortcut, treating the presence of message-related elements as sufficient evidence of persuasiveness. This is why we introduced a new rationale-focused method—dual-axis rationale supervision—combining evidence polarity (including vs. excluding evidence) and visual granularity (global vs. local evidence). This design encourages models to consider complementary visual perspectives when evaluating persuasiveness.

To further validate the benefits of dual-axis supervision, we conducted an additional ablation study comparing single-axis and dual-axis rationale supervision under matched training budgets. For each student backbone, we computed the average sequence length using its corresponding tokenizer. We kept batch size and gradient accumulation steps fixed while adjusting training steps according to differences in average sequence length, ensuring equal total training tokens across supervision settings.

We also controlled the number of training samples. Each single-axis setting contained 1,520 samples drawn from two relevant rationale categories (760 each), whereas dual-axis supervision contained 1,520 samples drawn equally from all four categories (380 each).

SupervisionPerspectiveQwenPhiGemmaLLaVA
Polarity-onlyGlobal73.766.470.268.2
Polarity-onlyLocal72.261.871.664.0
Granularity-onlyIncluding73.663.969.766.0
Granularity-onlyExcluding73.369.471.066.1
Dual-axisBoth75.470.972.668.6

From the table above, dual-axis supervision achieved the highest mean balanced accuracy across all four backbones, outperforming the best single-axis supervision by 1.7, 1.5, 1.0, and 0.4 percentage points on Qwen, Phi, Gemma, and LLaVA, respectively. These results suggest that combining complementary reasoning perspectives provides additional benefits beyond simply increasing the amount of rationale supervision. Detailed experimental settings and results have been added in Appendix E.4.

2. Classification Performance vs. Rationale Generation

The classification improvements over label-only SFT are relatively modest on some backbones. Nevertheless, we believe that reasoning quality is equally, if not more, important as it provides us with a lens to ensure the model uses faithful reasoning rather than relying on shortcuts. Hence, our objective is mainly to enable models to generate rationales while maintaining accurate predictions, not merely improve classification accuracy. Here, we show that label-only SFT significantly loses its reasoning ability over visual persuasion, losing an interpretive lens for the model's decision-making and making it difficult to judge whether the model has truly improved for this task or simply reinforced shortcuts. Specifically, we evaluated label-only SFT models across five random seeds by prompting them to generate rationales before making predictions.

StudentSFT (Direct) BASFT + Reasoning BAReasoning-SFT BA
Qwen74.669.175.1
Phi73.764.374.8
Gemma72.165.775.7
LLaVA72.965.872.5

Prompting SFT models to generate rationales reduced balanced accuracy by 5.5–9.4 percentage points across all four backbones. Furthermore, these models did not always produce the requested rationales. Qwen and LLaVA directly generated Yes/No predictions without rationales in 13.4% and 86.4% of cases, respectively, while Gemma exhibited relatively low rationale–prediction consistency (81.2 ± 9.74%).

AuthorsOpenReview ↗

W1. Limited Novelty and Statistical Reliability of Dual-Axis Supervision (2/2)

3. Statistical Reliability of Performance Improvements

To assess statistical reliability, we conducted paired bootstrap analyses comparing Reasoning-SFT against both SFT and SFT + Reasoning across five seeds. For each comparison, we preserved the pairing between methods within each seed and considered all 5^5 = 3,125 bootstrap resamples to estimate 95% confidence intervals (CIs) and p-values for BA differences.

StudentSFT vs. Reasoning-SFTSFT + Reasoning vs. Reasoning-SFT
Qwen[-0.40, 1.49], p = 0.123[1.03, 10.97], p = 0.028
Phi[0.01, 2.19], p = 0.038[5.53, 15.47], p=0.004
Gemma[0.17, 8.00], p = 0.042[5.03, 14.97], p=0.005
LLaVA[-2.81, 2.01], p = 0.676[1.73, 11.67], p=0.02

The classification improvements over direct SFT are backbone-dependent rather than uniformly significant. The analysis indicates significant improvements on Phi and Gemma (p < 0.05), whereas the confidence intervals for Qwen and LLaVA include zero. In contrast, Reasoning-SFT consistently outperforms SFT + Reasoning across all four backbones, with statistically significant improvements (p < 0.05) and confidence intervals entirely above zero. Although classification improvements over direct SFT are not statistically significant on some backbones, Reasoning-SFT offers a distinct practical advantage beyond improvements in direct classification accuracy, supporting explicit rationale generation without the substantial performance degradation observed in label-only SFT models.

We haved revised the Section 3.4.2 to clarify the objective and contributions of Reasoning-SFT and refine our claims regarding classification improvements to reflect their statistical uncertainty and backbone-dependent effects. Also, we have added statistical analysis results in Appendix E.5.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions (2/2)

4. Shortcut and Error Analyses

  • a) Message-relevance shortcut. Reviewer KFDA (Q1) asked how often models treat message-related content as sufficient evidence of persuasiveness and whether Reasoning-SFT reduces this shortcut. We measured its frequency among positive predictions and among false positives. On three of four backbones, both rates were lower under Reasoning-SFT than under Base-Reasoning, suggesting that its rationales relied less often on message relevance alone. We have defined the metrics in Appendix J and summarized the results in Section 3.1.

  • b) Rationale and prediction errors. Reviewer wPkk (Q2) asked how often rationale and prediction errors occur separately or together. We assessed each prediction against the ground-truth label and each rationale for groundedness and consistency. Across all four backbones, Reasoning-SFT gave correct answers with valid rationales more often than Base-Reasoning. Even when we considered only correct answers, Reasoning-SFT’s explanations were more often grounded in the image and consistent with the prediction. We have reported this analysis in Appendix I.3.

5. Evaluation Across Image Sources

Reviewer wPkk (Q3) asked whether the findings extend beyond AI-generated images. When trained and evaluated on web images, Reasoning-SFT outperformed label-only SFT on both Qwen and Phi. With mixed image sources, it outperformed label-only SFT on Qwen and matched it on Phi. These results suggest that Reasoning-SFT remains competitive with label-only SFT when training and evaluation include web-sourced images. We have reported both comparisons in Appendix M.

Reviewers’ Follow-Up Assessments

  • Reviewer wPkk confirmed that the rebuttal addressed their concerns about sensitivity, annotation procedures, error analysis, and generalization to real images and raised their rating to accept.
  • Reviewer WFfT maintained their positive rating.
  • Reviewer KFDA noted improved evaluation reliability and raised their rating to accept.

Strengths Highlighted by the Reviewers

  • a) Importance of visual persuasion reasoning. Reviewer wPkk (Strength 1) described visual persuasion as an underexplored problem that requires interpreting diverse visual cues in context.
  • b) Effectiveness of dual-axis supervision. Reviewer KFDA (Strength 1) noted competitive classification performance across MLLM families and a reduced “tendency to overpredict the persuasive class.”
  • c) Insights beyond classification accuracy. Reviewer KFDA (Strength 2) highlighted the finding that classification performance and rationale faithfulness can diverge, showing why both are worth evaluating.
  • d) An innovative framework for evaluating rationale faithfulness. Reviewer WFfT (Strength 1) described the framework, particularly its image-editing pipeline, as “an innovative way to audit MLLM reasoning.” Reviewer wPkk (Strength 2) highlighted how counterfactual edits test whether predictions respond to the visual evidence cited in a rationale.
  • e) Comprehensive and carefully controlled experiments. Reviewer wPkk (Strength 2) highlighted the range of models and training baselines. Reviewer WFfT (Strength 2) praised the care taken in dataset construction, model comparisons, metric validation, and experimental controls.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Summary of Revisions (1/2)

Dear Area Chair,

The reviewers' comments highlighted the need for stronger evidence supporting dual-axis rationale supervision and more rigorous validation of our teacher-generated rationales and faithfulness evaluation framework. In response, we conducted additional experiments and analyses to address these concerns and further examine the robustness and generalizability of our findings.

1. Dual-Axis Supervision

  • a) Matched-budget ablation. Reviewer KFDA (W1) questioned whether the gains came from combining complementary reasoning perspectives or from using more teacher-generated supervision, and Reviewer WFfT (W1) requested a single-axis control with a matched token budget. We compared dual-axis and single-axis supervision using the original teacher-generated rationales while matching the number of training examples and the total training tokens. Averaged over five seeds, dual-axis supervision outperformed on every backbone. We have reported this comparison in Section 3.4.2 and Appendix E.4.

  • b) Simple rationale-SFT. Reviewer wPkk (Q4, follow-up point 1) acknowledged the single-axis ablation and asked what the dual-axis structure adds beyond simple rationale-SFT without any reasoning structure. Reasoning-SFT outperformed a baseline trained on simple rationale-SFT baseline on every backbone, supporting the contribution of the proposed structure.

  • c) Classification with explanations. Reviewer wPkk (W1, follow-up point 2) noted that gains over direct label-only SFT were modest and nonsignificant on Qwen and LLaVA. However, asking label-only SFT to explain before predicting reduced its balanced accuracy, which shows decline in reasoning ability. Under this setting, Reasoning-SFT significantly outperformed label-only SFT on all four backbones. We have summarized the comparison in Section 3.4.2 and reported the statistical tests in Appendix E.5.

2. Faithfulness Framework Validation

  • a) Groundedness judge reliability. Reviewer wPkk (W2) and Reviewer WFfT (W2) questioned judge–human agreement. The revised judge improved agreement, and Reasoning-SFT retained the highest average groundedness. In response to Reviewer wPkk (follow-up point 3), we analyzed groundedness metric, and found most disagreements arose when the judge rejected a human-accepted rationale, supporting its use as a conservative screen tool. We have updated all groundedness results in Section 4.2.1 and Appendix I.1 and reported the validation in Section 4.1.1 and Appendix G.2.

  • b) Grounding of persuasive interpretations. Reviewer KFDA (W2) asked whether groundedness captures how visual cues support persuasive interpretations. In a new human study spanning literal, affective, symbolic, and compositional cues, annotators assessed both evidence presence and the interpretation drawn from it. The judge showed similar agreement with humans even when an appropriate interpretation was required. We have reported this study in Appendix G.7.

  • c) Sensitivity thresholds and image editors. Reviewer wPkk (W2) questioned whether sensitivity depended on near-zero confidence changes or the choice of editor. Excluding near-zero changes had little effect on scores, and an experiment with a second weak image editor preserved the method rankings on Qwen and Gemma. We have reported both checks in Section 4.2.3 and Appendices H.2–H.3.

  • d) Ambiguous examples. Reviewer WFfT (Q2) asked whether the findings held when human judgments of persuasiveness were evenly split. On Qwen and Gemma, Reasoning-SFT retained the highest groundedness and consistency, while sensitivity declined across methods. We have reported this analysis in Appendix I.4.

3. Teacher Rationale Quality

  • a) Post-hoc justification. Reviewer KFDA (W1, Q2) questioned whether teachers justified the ground truth label. When teachers predicted correctly without label, providing the label yielded similar explanations. Editing cited cues also reduced teacher confidence more often than editing irrelevant cues, linking the rationales to evidence relevant to the teachers’ predictions. We have summarized the similarity result in Section 3.3 and reported both tests in Appendix K.

  • b) Rationale quality control. Reviewer KFDA (W1) asked whether human-validated rationales improve classification. Annotators confirmed nearly all high G-Eval ratings and verified every GPT-5 correction. Balanced accuracy rose on two backbones and fell on two, suggesting that better explanations do not consistently yield better classifiers. We have reported this analysis in Appendix E.4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.