- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
The paper studies explainable hateful meme detection. The initial observation is that existing 'explain-then-detect' methods works on explanation generation followed by corresponding label prediction. Both the stages are optimized together. The authors argue that it causes task-objective interference and results worst performance than simple SFT.
Author proposed, ProKDA (Progressive Knowledge-to-Decision Alignment) that mimics how human annotators are trained. It learns the background, annotate, then re-review hard cases through a pipeline with one dedicated objective per stage:
- An agentic knowledge-construction step (GPT-4o agents + Wikipedia retrieval) builds a background-knowledge corpus for each meme.
- Stage 1 (SFT): It learns that background knowledge.
- Stage 2 (SFT): It adapts to generate a 'Yes' or 'No' in hatefulness detection.
- Confidence filtering + Stage 3 (DPO): It keeps only low-confidence “boundary” samples and refine the decision boundary.
The study attempts to show that decoupling them into separate stages improves the classification task.
Critical strengths
Strengths
- It has a clean ablation of stage ordering. Table 2 genuinely supports the compounding story, i.e., stage 1+2 > stage 2 alone, and adding Stage 3 helps further.
- The 'learn before you judge' framing is intuitive, and the ordered-SFT evidence is the paper’s most convincing part.
- It consists of a set of honest failure analyses. Figures 14 to 16 and the error section openly show ProKDA failing on highly implicit memes.
- The outputs are likely reproducible. The required LoRA config, LLaMA-Factory version, hardware, and full prompts are given.
Critical weaknesses
Weakness:
-
Check the claim of SFT Explain-then-Detect < SFT Direct Detection at line number 309. Prior works [1] highlighted it too. Please cite them. [1] EXPO-HM: LEARNING TO EXPLAIN-THEN-DETECT FOR HATEFUL MEME DETECTION
-
The real gap mentioned in the study is that the existing explain-then-detect methods combine (a) explanation generation and (b) label prediction in the same training process. It causes disturbance between the two task objectives that hurt both detection performance and explanation quality. The proposed method ProKDA is formulated around the interference hypothesis. The empirical gap is reported in Table 1 but the causal “interference” gap is not investigated enough. We need to be convinced that interference is the root cause behind the fall. Kindly add one or two gradient-conflict or representation-level analyses to endorse the claim.
-
Distillation vs. Decoupling: From Table 2, i.e., the Ablation study of ProKDA components, we see that
-
Stage 2 only (= Direct Detection SFT): W-F1 75.22
-
Stage 1+2: 81.79, i.e., adding the knowledge gives a gain of +6.57.
-
Full ProKDA (+DPO): 83.44, i.e., boundary alignment adds only +1.65
So, the teacher-synthesised external knowledge genuinely helps, but 1.65 from the “boundary alignment” piece that is most distinctive to the decoupling story contributes least. It seems that a large share of ProKDA’s advantage is due to teacher distillation, not progressive decoupling.
-
Narrow pipeline for knowledge quality analysis: It considers a total of 100 samples from HMC hateful memes and compares ProKDA’s knowledge via LLM judges and two human annotators on informativeness and consistency. But the knowledge base spans all three datasets and both classes. So, other datasets and knowledge related to benign memes remained unvalidated. Note that benign-meme knowledge matters for confounded cases.
In addition, the evaluation pipeline measures preference instead of correctness. Evaluators chose the better explanation or marked a tie, which is relative, but nothing in the pipeline verifies the knowledge is factually accurate or that the Wikipedia retrieval was on target. There is a high chance that hallucinated or wrong retrievals have propagated into the Stage-1 SFT (Eq. 4).
-
No variance, no significance testing: Many important deltas w.r.t. ProKDA are +1 to +4 points (Tables 1 and 2) with apparently single runs. Kindly include seeds/CIs/mean±std.
Questions for authors
Kindly consider the issues mentioned in the 'Weakness Section' to improve the work.
3 comments
Thank you for addressing the important points. I will raise my score.
Thank you for your reconsideration and feedback. We really appreciate it !!!
Dear Reviewer #n3XR,
We would like to kindly follow up on our rebuttal. We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns or if any issues remain unresolved.
Thank you for your time and consideration!