- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue that different samples in multimodal sarcasm detection require different analytical perspectives, which cannot be fully covered by predefined ones. It proposes a sample-adaptive dual-agent framework called ProCrit to achieve multi-perspective reasoning and revision, and presents a reciprocal training framework based on reinforcement learning. Extensive experiments were conducted on MMSD, MMSD2.0, and the cross-domain dataset RedEval to demonstrate the effectiveness of the framework.
Critical strengths
-
To address the issue that different samples in multimodal sarcasm detection require different analytical perspectives that cannot be fully covered by predefined ones, the authors propose ProCrit, a framework with sample-adaptive analytical perspectives, and train it using reinforcement learning. Extensive experiments were conducted on MMSD, MMSD2.0, and RedEval, achieving promising results.
-
Process-level reasoning trajectories are synthesized using GLM-4.6V, providing a feasible approach to constructing training data for multimodal sarcasm detection datasets lacking process-level annotations.
Critical weaknesses
-
Although this paper conducts extensive comparative experiments and detailed ablation studies, the framework introduces substantial computational overhead, while its performance remains weaker than some existing task-specific models on in-domain datasets. In addition, the full training experiments are mainly based on Qwen2.5-VL-7B, lacking validation on newer open-source models such as Qwen3-VL. It is suggested to further evaluate the framework on newer backbone models and analyze the trade-off between performance improvements and computational cost.
-
While this paper provides examples, human evaluations of perspective quality, and comparisons with fixed perspectives, but the sample adaptivity of the self-elicited perspectives lacks direct quantitative evidence. The differences between self-elicited and predefined perspectives in terms of adaptivity and diversity have not been sufficiently quantified.
Questions for authors
-
Can the Critic's score truly reflect the quality of the reasoning process? How can the authors prevent the Critic's score from being overly influenced by the correctness of the final prediction? Could they report its agreement with human evaluations of reasoning quality?
-
How can the correctness of process-level annotations be ensured? The process-level supervision in this paper was synthesized using GLM-4.6V and manually evaluated. However, for trajectories with correct final labels but incorrect reasoning steps, how can the authors prevent them from being treated as reliable supervision?
-
How does the model achieve truly adaptive perspective selection? Can the diversity of perspectives across samples and the contribution of each perspective be quantified?
4 comments
Using separate blind requests, GLM-5V-Turbo judged sample needs from image–text inputs and substantive perspective use () from anonymized reasoning across 100 label-balanced teacher trajectories and valid MMSD2.0 test examples:
| Trajectory source | Samples | Use when needed (%) | Use when not required (%) | Adaptivity (points) |
|---|---|---|---|---|
| Teacher supervision (training set) | 100 | 72.1 | 30.9 | 41.2 |
| Predefined: ProCrit (Fixed) Draft (test set) | 100 | 61.0 | 36.0 | 25.0 |
| Self-elicited: ProCrit Draft (test set) | 100 | 72.0 | 34.5 | 37.5 |
covers pragmatic intent/tone, expectations/norms, external background knowledge, and humor vs. sarcastic intent; uncertain need judgments are excluded per category.
Even when predefined roles receive credit for flexible content within fixed steps, ProCrit Draft raises needed-perspective coverage by +10.9 points (72.0% vs. 61.0%, matching the teacher’s 72.1%) and improves Adaptivity by +12.4 points over ProCrit (Fixed) Draft (37.5 vs. 25.0).
Diversity. Unlike predefined methods that apply an identical 3-step schema to every input, clustering generated step titles across all 2,409 MMSD2.0 and 1,004 RedEval test drafts (with all-MiniLM-L6-v2, average-linkage cosine distance 0.35) shows substantial variation in both reasoning length and perspective combinations:
| Metric | MMSD2.0 | RedEval |
|---|---|---|
| Step-count distribution: 1 / 2 / 3 / 4+ steps (%) | 0.0 / 14.2 / 81.4 / 4.4 | 0.1 / 21.3 / 73.8 / 4.8 |
| Mean steps: sarcastic / non-sarcastic | 2.98 / 2.84 | 2.96 / 2.75 |
| Perspective clusters appearing in samples | 32 | 19 |
| Distinct perspective-cluster combinations | 451 | 234 |
| Between-sample Jaccard distance (3-step drafts) | 0.616 | 0.627 |
Sarcastic samples elicit more steps on average, and the high Jaccard distances among 3-step drafts confirm that combination diversity persists at fixed reasoning length.
Contribution. We evaluate perspective contribution in both rationale quality and classification performance:
- Step-level quality (Table 4; Appendix D.1, Table 10). Across 100 label-balanced MMSD2.0 test examples rated blindly by five annotators (0–5 scale), self-elicited perspectives achieve higher relevance, unique contribution, and necessity while nearly halving redundancy:
| Method | Relevance ↑ | Unique contribution ↑ | Redundancy ↓ | Unsupported ↓ | Necessity ↑ |
|---|---|---|---|---|---|
| Fixed-role Reasoner | 3.34 | 4.00 | 0.91 | 2.82 | 3.77 |
| ProCrit Draft | 4.04 | 4.37 | 0.52 | 1.51 | 4.07 |
| ProCrit Revise | 4.17 | 4.52 | 0.52 | 1.80 | 4.42 |
- Task-level performance (Tables 1, 5, and 25). Under standard SFT+GRPO, Dynamic-role Reasoner exceeds Fixed-role Reasoner by +1.0, +0.5, and +2.1 F1 on MMSD2.0, MMSD, and RedEval. With identical critic and revision training, adaptive ProCrit still outperforms ProCrit (Fixed) by +0.93 F1 on both MMSD2.0 and RedEval.
We have included these analyses in Appendix D.13 (Tables 26–27), with the relevant content highlighted in blue.
Q1: Critic score validity and agreement with human evaluation
Reply:
The Critic is prevented from collapsing into a label-correctness proxy by three coupled designs: (1) SFT initializes evaluation from a 0–2 process rubric assessing interpretation accuracy, cross-modal reasoning, and coherence, explicitly instructing that a correct conclusion does not guarantee a high score (Prompt 6, Appendix C.3); (2) in RL, while anchors scores to verifiable correctness, <score> is generated conditioned on <feedback>, which rewards only when it induces an actual error-fixing revision. Empirically, even though Proposal GRPO already includes binary label accuracy , removing the Critic score reward still drops MMSD2.0 Draft F1 from 81.1 to 79.9 (Table 17), showing that the score provides process-level signal beyond answer correctness.
Agreement with human evaluation. We matched the Critic’s scores () against the five-annotator blind ratings (0–5 scale) on the 100 label-balanced MMSD2.0 ProCrit Drafts in Table 4 (Appendix D.1, Table 9), using per-example mean ratings and tie-corrected Spearman rank correlation:
| Human-rated dimension | Spearman with Critic | -value |
|---|---|---|
| Overall quality (↑) | 0.36 | < 0.001 |
| Evidence grounding (↑) | 0.34 | < 0.001 |
| Logical coherence (↑) | 0.33 | < 0.001 |
| Rationale–label consistency (↑) | 0.30 | 0.002 |
| Core-cue coverage (↑) | 0.30 | 0.003 |
| Hallucination severity (↓) | −0.35 | < 0.001 |
Under the same rank-correlation metric, binary prediction correctness alone yields () with human Overall quality, whereas the Critic score reaches , capturing reasoning quality variance beyond answer correctness.
Discrimination within correct predictions. Across all 2,409 MMSD2.0 test drafts, the Critic withholds the top score () on 13.3% (265 / 1,987) of correct predictions (: 131; : 134; : 1,722). Within the 83 correct predictions in the 100-sample human-evaluated set, human quality ratings rise monotonically with (, ):
| Critic score | Evidence grounding (↑) | Hallucination (↓) | Human Overall (↑) | |
|---|---|---|---|---|
| 0 | 6 | 1.37 | 1.80 | 0.80 |
| 1 | 6 | 2.30 | 1.13 | 1.80 |
| 2 | 71 | 3.17 | 0.73 | 2.90 |
Even when final labels are identical and correct, drafts scored exceed those scored by +1.60 points in human Overall quality (2.90 vs. 1.30) with less than half the hallucination severity, confirming that the Critic retains sensitivity to reasoning quality within the same correctness level (see also Figure 8, Appendix E). We have added this analysis to Appendix D.1.
Q2: Quality and reliability of process-level supervision
Reply:
ProCrit promotes reliable process supervision through protocol-constrained synthesis and post-hoc label verification, without relying on final-label correctness alone. We first quantify the incidence of flawed steps in our synthesized supervision through human evaluation, and then summarize how ProCrit safeguards reasoning quality across data synthesis, reinforcement learning, and inference.
Human audit of label-correct trajectories. In our blind human evaluation of 100 label-balanced, label-correct GLM-4.6V training trajectories (Appendix D.1, Table 9), we group trajectories by their per-example mean ratings into three categories:
| Category ( label-correct trajectories) | Count | Evidence grounding ↑ | Logical coherence ↑ | Cue coverage ↑ | Hallucination ↓ | Overall ↑ |
|---|---|---|---|---|---|---|
| Sound reasoning () | 77 | 4.86 | 4.79 | 4.77 | 0.05 | 4.75 |
| Grounded but incomplete cues () | 14 | 2.10 | 2.23 | 0.33 | 0.29 | 0.36 |
| Flawed / hallucinated steps () | 9 | 0.27 | 0.34 | 0.11 | 3.79 | 0.14 |
| All audited trajectories (Table 9) | 100 | 4.06 | 4.03 | 3.73 | 0.42 | 3.72 |
Overall, 91% of label-correct trajectories remain factually grounded (), including 77% with thorough multi-perspective reasoning and 14% with valid observations that miss secondary cues, while 9% contain hallucinated or flawed intermediate steps despite reaching the correct label.
Safeguards across synthesis, training, and inference. Across the pipeline, three stages keep step-level errors low during synthesis and prevent the remaining 9% from degrading the final model:
- Protocol-constrained, label-blind synthesis and repair. During rollout, GLM-4.6V must contribute a distinct, context-conditioned perspective at each step without access to the gold label, which is verified only afterward to avoid post-hoc rationalization. Failed rollouts are regenerated using critic feedback rather than blind retries (Appendix B.1).
- Process-level differentiation during RL. SFT serves only as initialization. During GRPO, the critic’s quality score differentiates candidate drafts sharing the same correct prediction and assigns lower relative advantages to weaker rationales. Consistent with this design, RL improves MMSD2.0 Draft F1 from 78.1 to 81.1 over SFT alone (Table 2; Figure 9), and removing the critic evaluation reward lowers Draft F1 to 79.9 (Table 17).
- External critique at inference. At test time, drafts with weak reasoning—including correct predictions scored below 2 by the critic—are revised under targeted feedback, further reducing human-rated hallucination from 0.95 to 0.84 (Table 4; Figure 8).
We have added this analysis in Appendices B.2 and D.1.
Thank you again for your time and for your review. With the discussion period nearing its end, we would welcome any feedback on our response, especially if there are concerns that you feel still need to be addressed. We would be happy to follow up on those points.
Best regards,
The Authors
We greatly appreciate the reviewer’s valuable comments and suggestions, which have helped us improve the clarity and quality of our manuscript. We address each point below.
W1: Newer backbone validation and performance–cost trade-off
Reply:
We have added experiments with Qwen3-VL-8B-Instruct, completing SFT and GRPO for the ProCrit proposal and three Reasoner baselines, and evaluating them alongside their zero-shot counterparts on MMSD2.0 and out-of-domain RedEval. ProCrit retains the existing frozen Qwen2.5-VL-7B critic; this comparison therefore evaluates adaptation to a newer proposal backbone. We follow the experimental setup used in our main experiments. The results are reported as Macro-F1 and Accuracy (%); Macro-F1 averages the F1 scores of the sarcastic and non-sarcastic classes.
Perspective design. On Qwen3, all three trained Reasoners outperform their zero-shot counterparts. Dynamic-role also outperforms both trained alternatives, exceeding Fixed-role by 1.1 and 1.9 points on MMSD2.0 and RedEval, respectively. These results support the usefulness of learned adaptive perspectives beyond the original backbone.
ProCrit training and revision. Relative to Dynamic-role, ProCrit Draft improves RedEval F1 by 0.4 points without an inference-time critic call, supporting cross-domain generalization through single-response inference. On MMSD2.0, ProCrit's advantage emerges after revision: F1 rises from 82.2 to 83.7, exceeding Dynamic-role by 0.9 points. ProCrit thus benefits initial prediction on RedEval and feedback-guided refinement on MMSD2.0, with different benefits across domains.
Performance–cost trade-off.
Generalization and reasoning capabilities relative to task-specific discriminative baselines. The compared task-specific discriminative baselines primarily learn or enrich multimodal features for label prediction. ProCrit further incurs autoregressive generation to produce explicit, sample-adaptive reasoning that can be inspected and corrected, which is important for practical applications requiring human oversight. As shown in Table 13 (Appendix D.3), ProCrit draft achieves near-best cross-domain performance on RedEval, outperforming other discriminative baselines by 4.1–17.1 F1 points. These results demonstrate that ProCrit's benefits extend beyond in-domain accuracy to cross-domain generalization and explicit, revisable reasoning.
Training and inference costs. Proposal training takes approximately 21.5 hours on eight GPUs, while single-response inference takes approximately 4.0 seconds per example. However, the trained critic can be reused across compatible reasoning backbones, as demonstrated above, and revision is optional, allowing users to balance performance requirements and computational budget.
We have added the Qwen3 results and analysis in Appendix D.12 (Table 25) .
W2 & Q3: Quantifying perspective adaptivity, diversity, and contribution
Reply:
Through training on dynamic-role trajectories (§3.1), ProCrit internalizes adaptive perspective selection into a single autoregressive sequence: each step is conditioned on the multimodal input and all preceding findings until the accumulated evidence suffices, while dual-stage GRPO and critic-guided revision (§3.3) reinforce perspectives that capture decisive sarcasm cues and recover omitted angles. Below, we quantify its adaptivity, diversity, and contribution relative to predefined perspectives.
Adaptivity. Adaptivity measures whether perspective selection responds to sample-specific needs. We quantify it as the average selection-rate gap between needed (Np=1) and non-required (Np=0) cases:
\frac{100}{|\mathcal{P}|} \sum_{p\in\mathcal{P}} \left[ \widehat{P}(S_p=1\mid N_p=1) - \widehat{P}(S_p=1\mid N_p=0) \right].$$