- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Reconstruct-then-Evolve, a data-centric framework for harmful meme detection that addresses weak cross-modal grounding and limited diversity in existing training datasets. The framework first reconstructs memes by identifying their core harmful semantics and image–text relationships, then uses verifier-guided Monte Carlo Tree Search (MCTS) to generate diverse realizations through controlled changes in expression strategies, scenarios, templates, and visual styles. A multi-dimensional verifier filters synthetic examples based on visual quality, semantic fidelity, and cross-modal contribution, and accepted harmful examples are paired with benign counterparts for detector fine-tuning. Experiments across four benchmarks (FHM, MAMI, HarMeme, and PrideMM) demonstrate consistent improvements over standard fine-tuning and competitive existing methods. Additional ablations, search strategy comparisons, human evaluations, and transfer experiments across detector backbones and frameworks further support the effectiveness of the proposed approach.
Critical strengths
-
The proposed framework consistently improves detection accuracy across four benchmarks, with gains of 5.1–10.7 percentage points over vanilla fine-tuning. Improvements also transfer across different detector backbones and existing frameworks, supporting its practical value.
-
The paper includes well-designed ablations, matched-sample comparisons of MCTS against random and beam search, scaling experiments, and human evaluation of synthetic data quality. These experiments provide convincing evidence for the effectiveness of the proposed pipeline and its individual components.
Critical weaknesses
-
While Reconstruct substantially improves detection performance, it remains unclear whether the gains result from better cross-modal grounding or simply making harmful content more explicit and easier to classify. Additional modality-specific analyses would help substantiate the proposed mechanism.
-
Although MCTS is compared fairly against random and beam search, the paper lacks a comparably sized conventional data augmentation baseline. Moreover, the main FHM evaluation uses the test-seen split, leaving generalization to genuinely unseen meme distributions less thoroughly established.
Questions for authors
-
Can the authors clarify whether the gains primarily come from improved image–text grounding or from making harmful content more explicit and easier to recognize? An analysis separating samples that require grounding repairs from those that do not would help substantiate the claimed mechanism.
-
While the paper compares MCTS with random and beam search at matched sample counts, could the authors clarify whether the full pipeline outperforms simpler augmentation methods under comparable data and training budgets?
-
Given that the main FHM experiments use the test-seen split, could the authors report results on test-unseen or provide additional evidence that the improvements extend beyond familiar meme distributions?
2 comments
Response to m8KX(2/2): Augmentation Budgets and Generalization
W2/Q2: Augmentation under comparable data and search budgets.
We report two complementary controls: matching the accepted training data and matching the candidate-generation budget.
Matched data and training. Section 4.2/Figure 3 compares search strategies using the same seeds, mutation operators, generator, verifier, and acceptance rules. At 1.0x on FHM, MCTS reaches 85.37 ± 0.31% accuracy, versus 82.00 ± 0.39% for random and 82.80 ± 0.37% for beam search (three-seed means ± SD).
The additional augmentation comparison uses the same reconstructed base of 8,500 examples and adds 3,050 harmful + 3,050 benign examples per method. All conditions therefore train on 14,600 entries with identical initialization, optimizer, and schedule: three epochs, learning rate 5e-5, effective batch size 32, and 1,371 optimizer updates.
| Augmentation | Accuracy (%) | Macro-F1 (%) |
|---|---|---|
| Class-matched repetition | 82.10 | 81.90 |
| Independent one-shot synthesis | 82.30 | 82.12 |
| Diversity selection | 83.10 | 82.97 |
| Random search | 82.00 | 81.80 |
| Greedy search | 83.00 | 82.86 |
| Beam search, width 64 | 83.10 | 82.98 |
| MCTS Evolve | 85.30 | 85.22 |
These are single-checkpoint results on the same 1,000-example FHM test-seen set. One-shot synthesis generates independently from seed specifications. Diversity selection uses fixed-embedding farthest-point sampling from the same qualifying one-shot pool as its uniform-selection comparator. Search-based methods each supply 3,050 distinct harmful examples sampled from their complete qualifying pools; Figure 3 instead uses generation-order prefixes and three-seed statistics.
With training data size and updates fixed, MCTS improves accuracy over repetition, one-shot synthesis, and diversity selection by 3.20, 3.00, and 2.20 percentage points. This comparison measures the additional value of Evolve on a common reconstructed base.
Matched candidate-generation budget. With 25,091 candidate proposals for each search strategy, the qualifying pools and corresponding detector accuracies are:
| Search | Qualifying candidates | Approx. scale | Accuracy (%) |
|---|---|---|---|
| Random | 4,256 | 1.40x | 81.7 |
| Greedy | 4,968 | 1.63x | 82.5 |
| Beam | 5,347 | 1.75x | 82.6 |
| MCTS | 6,122 | 2.00x | 84.9 |
Here, 1.0x corresponds to 3,050 qualifying harmful examples; pool sizes vary with search yield. MCTS produces 43.84% more qualifying candidates than random search and improves accuracy by 3.2, 2.4, and 2.3 points over random, greedy, and beam search. Together, the two controls show that MCTS produces more effective training data at a fixed accepted count and higher accuracy within a fixed candidate-generation budget.
We will extend the augmentation comparison to label-preserving caption paraphrasing/back-translation, mild image transformations, and their combination. Each will add 3,050 harmful + 3,050 benign examples under matched training updates, evaluated with both original and reconstructed bases.
W2/Q3: Test-unseen and cross-dataset generalization.
We evaluated the official FHM test-unseen split during our experiments. We selected test-seen for the main evaluation to enable direct comparison with a wider range of prior work using the same split. Our Qwen2.5-VL-7B test-unseen results are:
| Training method | Test-unseen accuracy (%) |
|---|---|
| Reconstruct | 80.80 |
| Reconstruct-then-Evolve | 84.80 |
Evolve provides a further 4.00-point improvement over Reconstruct on test-unseen.
The paper also evaluates cross-dataset OOD generalization in Section 4.3/Figure 5(c) and Appendix I/Table I.1. With FHM as the training source, the results are:
| FHM-source training | MultiOFF Acc/Macro-F1 (%) | MAMI Acc/Macro-F1 (%) |
|---|---|---|
| Vanilla SFT | 63.09/58.40 | 66.20/65.36 |
| Reconstruct | 64.43/61.67 | 69.10/68.98 |
| Reconstruct-then-Evolve | 67.79/65.42 | 72.70/72.61 |
Over Vanilla SFT, the full pipeline gains 4.70 accuracy and 7.02 macro-F1 points on MultiOFF, and 6.50 and 7.25 points on MAMI. These results demonstrate improvements on the official test-unseen split and on the cross-dataset OOD evaluations already included in the paper.
We will incorporate these results, clarifications, and experimental details into the revised paper and appendix. Thank you again for your constructive feedback; we welcome any further suggestions.
Response to m8KX(1/2): Reconstruction and Cross-Modal Grounding
Thank you for recognizing the consistent improvements, controlled comparisons, and transfer results. Your questions help us clarify the mechanism of Reconstruct and the evidence for its generalization.
W1/Q1: Recovering the original harmful image-text relation.
Reconstruct aims to recover the visual realization of the harmful meaning already expressed by a source meme through minimal content changes. Its semantic anchor, comprising the target, attributed property, harm category, and core proposition, remains fixed. When visual support for that meaning is missing, ambiguous, or incorrectly depicted, Reconstruct repairs the relevant referent, relation, or evidence while retaining the original caption.
Appendix J.1 specifies selective, single-link repair: already-grounded (
strong) sources receive no semantic repair, while weakly grounded sources allow only the diagnosed change. Instructions preserve other content and prohibit new targets, claims, slurs, or stronger hostility. If source semantics cannot be preserved, no repair is applied. Evolve then expands alternative expressions of the fixed proposition.Grounding versus stronger unimodal cues. Strengthening harmful cues in only one modality may encourage shortcut learning. We assess the role of grounding through reconstruction ablations, followed by caption-controlled and modality-specific tests. Section 4.4/Figure 6(a) reports the following FHM results:
Naive Generation preserves coarse harmful intent while allowing expression changes; Ungrounded Reconstruction preserves source appearance without explicit grounding repair; Caption Distillation adds textual descriptions. Grounded Reconstruct improves accuracy over these alternatives by 4.30, 1.90, and 3.40 percentage points, respectively. The gain over Ungrounded Reconstruction supports the contribution of grounding repair.
Controlling caption explicitness. We further evaluate 100 unmodified memes selected from the official FHM test-seen and test-unseen splits that require joint interpretation (50 harmful/50 benign). Each condition uses one prespecified checkpoint and threshold.
The two explicit-caption controls use identical rewritten captions for the same 3,050 harmful training items, the same 5,450 benign items, and matched rendering/training settings. D2/D3 scores are recorded without acceptance thresholds, allowing generic imagery to enter the comparison. Holding caption wording fixed, claim-specific visual evidence improves accuracy from 74% to 79%. Reconstruct also reaches 79% while retaining the original captions. These results connect the improvement to the visual evidence supporting the image-text relation.
Modality and relation diagnostics. We compare Direct Evolve (MCTS initialized from original memes), no-D3 (cross-modal scoring removed from both reward and acceptance), and Full at matched accepted counts and training budgets. The table reports accuracy (%) averaged over three training seeds.
Full's largest gains occur with both modalities available. They persist when embedded captions are removed and supplied separately: this condition uses exactly the cleaned images used for image-only evaluation. The separate triple diagnostic contains 60 harmful anchors, each paired with an image-changed and a text-changed benign counterpart. A triple counts as correct only when all three predictions are correct. Full improves this accuracy by 13.89 points over no-D3 and 18.89 over Direct Evolve, supporting greater sensitivity to image-text relations.
Sources requiring repair. Appendix J.1 already distinguishes
strongfrom repair-needed sources before generation. We will add a stratified comparison of targeted repair and source-faithful re-rendering under matched data and training budgets. This analysis will quantify the contribution of repairing missing visual premises within each source category.