acceptodds
Under review as a conference paper at ICLR 2027

Reconstruct, Then Evolve: Grounded Data Synthesis for Harmful Meme Detection

Abstract

Harmful memes often convey abuse through implicit interactions between images and text, making reliable detection critically dependent on training data that both faithfully captures harmful semantics and covers diverse ways those semantics can be expressed. Existing work has largely improved model architectures, contextual reasoning, or inference procedures while leaving the training distribution unchanged. This leaves two data bottlenecks: Weak cross-modal grounding within individual examples and limited coverage of harmful realizations in finite training sets. Effective synthesis therefore requires preserving *what makes a meme harmful* while systematically varying *how that meaning is realized*. We introduce *Reconstruct-then-Evolve*, a data-centric framework that first reconstructs source memes into semantically grounded realizations and then expands them through verifier-guided Monte Carlo Tree Search over alternative expression strategies, scenes, templates, and visual styles. Without modifying the detector architecture or inference procedure, our synthesized data raise accuracy by 5.1-10.7 percentage points across four benchmarks and outperform prior published baselines on all four. The gains persist across multiple detector backbones, with improvements of 7.4-10.7 percentage points over fine-tuning on the original training data. These results show that systematically improving the training distribution provides a powerful and complementary path toward more robust harmful meme detection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Harmful Meme Detection, Multimodal Learning, Synthetic Data, Data-Centric Learning, Monte Carlo Tree Search, Multimodal Safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m8KXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Reconstruct-then-Evolve, a data-centric framework for harmful meme detection that addresses weak cross-modal grounding and limited diversity in existing training datasets. The framework first reconstructs memes by identifying their core harmful semantics and image–text relationships, then uses verifier-guided Monte Carlo Tree Search (MCTS) to generate diverse realizations through controlled changes in expression strategies, scenarios, templates, and visual styles. A multi-dimensional verifier filters synthetic examples based on visual quality, semantic fidelity, and cross-modal contribution, and accepted harmful examples are paired with benign counterparts for detector fine-tuning. Experiments across four benchmarks (FHM, MAMI, HarMeme, and PrideMM) demonstrate consistent improvements over standard fine-tuning and competitive existing methods. Additional ablations, search strategy comparisons, human evaluations, and transfer experiments across detector backbones and frameworks further support the effectiveness of the proposed approach.

Critical strengths
  1. The proposed framework consistently improves detection accuracy across four benchmarks, with gains of 5.1–10.7 percentage points over vanilla fine-tuning. Improvements also transfer across different detector backbones and existing frameworks, supporting its practical value.

  2. The paper includes well-designed ablations, matched-sample comparisons of MCTS against random and beam search, scaling experiments, and human evaluation of synthetic data quality. These experiments provide convincing evidence for the effectiveness of the proposed pipeline and its individual components.

Critical weaknesses
  1. While Reconstruct substantially improves detection performance, it remains unclear whether the gains result from better cross-modal grounding or simply making harmful content more explicit and easier to classify. Additional modality-specific analyses would help substantiate the proposed mechanism.

  2. Although MCTS is compared fairly against random and beam search, the paper lacks a comparably sized conventional data augmentation baseline. Moreover, the main FHM evaluation uses the test-seen split, leaving generalization to genuinely unseen meme distributions less thoroughly established.

Questions for authors
  1. Can the authors clarify whether the gains primarily come from improved image–text grounding or from making harmful content more explicit and easier to recognize? An analysis separating samples that require grounding repairs from those that do not would help substantiate the claimed mechanism.

  2. While the paper compares MCTS with random and beam search at matched sample counts, could the authors clarify whether the full pipeline outperforms simpler augmentation methods under comparable data and training budgets?

  3. Given that the main FHM experiments use the test-seen split, could the authors report results on test-unseen or provide additional evidence that the improvements extend beyond familiar meme distributions?

2 comments
AuthorsOpenReview ↗

Response to m8KX(1/2): Reconstruction and Cross-Modal Grounding

Thank you for recognizing the consistent improvements, controlled comparisons, and transfer results. Your questions help us clarify the mechanism of Reconstruct and the evidence for its generalization.

W1/Q1: Recovering the original harmful image-text relation.

Reconstruct aims to recover the visual realization of the harmful meaning already expressed by a source meme through minimal content changes. Its semantic anchor, comprising the target, attributed property, harm category, and core proposition, remains fixed. When visual support for that meaning is missing, ambiguous, or incorrectly depicted, Reconstruct repairs the relevant referent, relation, or evidence while retaining the original caption.

Appendix J.1 specifies selective, single-link repair: already-grounded (strong) sources receive no semantic repair, while weakly grounded sources allow only the diagnosed change. Instructions preserve other content and prohibit new targets, claims, slurs, or stronger hostility. If source semantics cannot be preserved, no repair is applied. Evolve then expands alternative expressions of the fixed proposition.

Grounding versus stronger unimodal cues. Strengthening harmful cues in only one modality may encourage shortcut learning. We assess the role of grounding through reconstruction ablations, followed by caption-controlled and modality-specific tests. Section 4.4/Figure 6(a) reports the following FHM results:

Training conditionAccuracy (%)Macro-F1 (%)
Vanilla SFT74.8073.75
Caption Distillation78.1077.81
Naive Generation77.2076.78
Ungrounded Reconstruction79.6079.42
Grounded Reconstruct81.5081.38

Naive Generation preserves coarse harmful intent while allowing expression changes; Ungrounded Reconstruction preserves source appearance without explicit grounding repair; Caption Distillation adds textual descriptions. Grounded Reconstruct improves accuracy over these alternatives by 4.30, 1.90, and 3.40 percentage points, respectively. The gain over Ungrounded Reconstruction supports the contribution of grounding repair.

Controlling caption explicitness. We further evaluate 100 unmodified memes selected from the official FHM test-seen and test-unseen splits that require joint interpretation (50 harmful/50 benign). Each condition uses one prespecified checkpoint and threshold.

Training conditionCorrect/100
Vanilla SFT71
Explicit captions + generic imagery74
Identical explicit captions + claim-specific visual evidence79
Reconstruct with original captions79

The two explicit-caption controls use identical rewritten captions for the same 3,050 harmful training items, the same 5,450 benign items, and matched rendering/training settings. D2/D3 scores are recorded without acceptance thresholds, allowing generic imagery to enter the comparison. Holding caption wording fixed, claim-specific visual evidence improves accuracy from 74% to 79%. Reconstruct also reaches 79% while retaining the original captions. These results connect the improvement to the visual evidence supporting the image-text relation.

Modality and relation diagnostics. We compare Direct Evolve (MCTS initialized from original memes), no-D3 (cross-modal scoring removed from both reward and acceptance), and Full at matched accepted counts and training budgets. The table reports accuracy (%) averaged over three training seeds.

DiagnosticDirect Evolveno-D3Full
Joint-interpretation subset: multimodal74.0077.0083.33
Text only64.0065.0065.33
Caption-removed image only58.6759.6760.00
Cleaned image + original text73.6776.6783.00
Separate triple set: all three correct52.7857.7871.67

Full's largest gains occur with both modalities available. They persist when embedded captions are removed and supplied separately: this condition uses exactly the cleaned images used for image-only evaluation. The separate triple diagnostic contains 60 harmful anchors, each paired with an image-changed and a text-changed benign counterpart. A triple counts as correct only when all three predictions are correct. Full improves this accuracy by 13.89 points over no-D3 and 18.89 over Direct Evolve, supporting greater sensitivity to image-text relations.

Sources requiring repair. Appendix J.1 already distinguishes strong from repair-needed sources before generation. We will add a stratified comparison of targeted repair and source-faithful re-rendering under matched data and training budgets. This analysis will quantify the contribution of repairing missing visual premises within each source category.

AuthorsOpenReview ↗

Response to m8KX(2/2): Augmentation Budgets and Generalization

W2/Q2: Augmentation under comparable data and search budgets.

We report two complementary controls: matching the accepted training data and matching the candidate-generation budget.

Matched data and training. Section 4.2/Figure 3 compares search strategies using the same seeds, mutation operators, generator, verifier, and acceptance rules. At 1.0x on FHM, MCTS reaches 85.37 ± 0.31% accuracy, versus 82.00 ± 0.39% for random and 82.80 ± 0.37% for beam search (three-seed means ± SD).

The additional augmentation comparison uses the same reconstructed base of 8,500 examples and adds 3,050 harmful + 3,050 benign examples per method. All conditions therefore train on 14,600 entries with identical initialization, optimizer, and schedule: three epochs, learning rate 5e-5, effective batch size 32, and 1,371 optimizer updates.

AugmentationAccuracy (%)Macro-F1 (%)
Class-matched repetition82.1081.90
Independent one-shot synthesis82.3082.12
Diversity selection83.1082.97
Random search82.0081.80
Greedy search83.0082.86
Beam search, width 6483.1082.98
MCTS Evolve85.3085.22

These are single-checkpoint results on the same 1,000-example FHM test-seen set. One-shot synthesis generates independently from seed specifications. Diversity selection uses fixed-embedding farthest-point sampling from the same qualifying one-shot pool as its uniform-selection comparator. Search-based methods each supply 3,050 distinct harmful examples sampled from their complete qualifying pools; Figure 3 instead uses generation-order prefixes and three-seed statistics.

With training data size and updates fixed, MCTS improves accuracy over repetition, one-shot synthesis, and diversity selection by 3.20, 3.00, and 2.20 percentage points. This comparison measures the additional value of Evolve on a common reconstructed base.

Matched candidate-generation budget. With 25,091 candidate proposals for each search strategy, the qualifying pools and corresponding detector accuracies are:

SearchQualifying candidatesApprox. scaleAccuracy (%)
Random4,2561.40x81.7
Greedy4,9681.63x82.5
Beam5,3471.75x82.6
MCTS6,1222.00x84.9

Here, 1.0x corresponds to 3,050 qualifying harmful examples; pool sizes vary with search yield. MCTS produces 43.84% more qualifying candidates than random search and improves accuracy by 3.2, 2.4, and 2.3 points over random, greedy, and beam search. Together, the two controls show that MCTS produces more effective training data at a fixed accepted count and higher accuracy within a fixed candidate-generation budget.

We will extend the augmentation comparison to label-preserving caption paraphrasing/back-translation, mild image transformations, and their combination. Each will add 3,050 harmful + 3,050 benign examples under matched training updates, evaluated with both original and reconstructed bases.

W2/Q3: Test-unseen and cross-dataset generalization.

We evaluated the official FHM test-unseen split during our experiments. We selected test-seen for the main evaluation to enable direct comparison with a wider range of prior work using the same split. Our Qwen2.5-VL-7B test-unseen results are:

Training methodTest-unseen accuracy (%)
Reconstruct80.80
Reconstruct-then-Evolve84.80

Evolve provides a further 4.00-point improvement over Reconstruct on test-unseen.

The paper also evaluates cross-dataset OOD generalization in Section 4.3/Figure 5(c) and Appendix I/Table I.1. With FHM as the training source, the results are:

FHM-source trainingMultiOFF Acc/Macro-F1 (%)MAMI Acc/Macro-F1 (%)
Vanilla SFT63.09/58.4066.20/65.36
Reconstruct64.43/61.6769.10/68.98
Reconstruct-then-Evolve67.79/65.4272.70/72.61

Over Vanilla SFT, the full pipeline gains 4.70 accuracy and 7.02 macro-F1 points on MultiOFF, and 6.50 and 7.25 points on MAMI. These results demonstrate improvements on the official test-unseen split and on the cross-dataset OOD evaluations already included in the paper.

We will incorporate these results, clarifications, and experimental details into the revised paper and appendix. Thank you again for your constructive feedback; we welcome any further suggestions.

Reviewer_NdhkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reconstruct-then-Evolve, a data synthesis framework for harmful meme detection. Reconstruct recovers a semantic anchor and an editable realization from a source meme, then renders a reconstructed example. Evolve uses verifier-guided Monte Carlo Tree Search to explore alternative expression strategies, scenarios, templates, and visual styles while preserving the semantic anchor. Generated candidates are evaluated for realization quality, harmful-chain fidelity, and cross-modal contribution, and accepted harmful examples are paired with benign counterparts. The framework is evaluated on four benchmarks, with additional backbone and framework transfer experiments, cross-dataset evaluation, ablations, matched-count search comparisons, and human assessment. The results demonstrate useful improvements in downstream classification.

Critical strengths
  1. Consistent empirical utility across several settings. The framework improves results on four benchmarks and transfers across detector backbones and detection frameworks. Reconstruct improves performance without increasing the training sample count. These results provide meaningful evidence that the construction of training examples matters beyond simply enlarging the dataset.

  2. Useful comparisons against alternative synthesis and search strategies. The reconstruction ablations include naive generation, ungrounded reconstruction, and caption distillation. The search experiments compare MCTS with random and beam search using the same accepted sample count and shared synthesis components. These comparisons substantiate the practical value of several design choices and strengthen the empirical contribution.

Critical weaknesses
  1. The evidence does not sufficiently distinguish improved cross-modal semantic learning from other benefits of data synthesis. Aggregate classification improvements, transfer results, and expression-coverage analyses establish useful training data, but do not isolate whether the detector learns more reliable image–text relations rather than benefiting from broader coverage or stronger unimodal cues. The D3 ablation is relevant, but removes D3 from both search reward and candidate acceptance. Consequently, its performance drop establishes the utility of the combined D3 intervention rather than identifying the mechanism responsible for the improvement. Moreover, the D3 rubric permits mutually reinforcing modalities without requiring that the harmful interpretation depend on their combination. The human assessment provides supporting evidence, but does not fully resolve this issue. Appendix B reports semantic-preservation and overall validity scores, including score distributions and validity rates. However, the evolved-sample results are not separated by verification dimension, and no inter-rater agreement statistic is reported. Rounded mean scores can conceal disagreement about whether semantics and cross-modal contribution are preserved. These limitations leave the central cross-modal interpretation less firmly supported than the downstream effectiveness claim.

  2. The added value of coupling reconstruction with controlled search remains insufficiently isolated. The existing ablations support individual choices, and the matched-count search comparisons demonstrate an advantage for MCTS. However, they do not fully establish how much of the benefit depends on reconstructing the source examples before evolution, rather than applying controlled, verifier-guided evolution directly to the original examples. A comparison that isolates this coupling would help determine whether the two-stage design addresses a limitation that a simpler pipeline cannot adequately handle. Since the contribution rests substantially on this integration, the remaining uncertainty matters to my assessment of its methodological significance.

Questions for authors
  1. Clarification and supporting evidence: Can the authors provide more direct evidence of improved image–text relation sensitivity, such as results on multimodal confounder pairs or a targeted subset requiring joint interpretation? Text-only and image-only diagnostics could also help, provided that the latter removes the embedded caption to avoid OCR leakage.

  2. Clarification of attribution: Were the D3 ablations matched in accepted training sample count and training steps? Can the effects of using D3 for search reward and for final acceptance be separated? Is there a comparison between Evolve initialized from original examples and Evolve initialized from reconstructed examples under otherwise matched conditions?

  3. Clarification of human evaluation: Were D1, D2, and D3 rated separately for evolved samples? If so, please report dimension-specific validity rates, failure categories, and agreement computed from the individual ratings before averaging or rounding, preferably using ordinal Krippendorff’s alpha.

6 comments
AuthorsOpenReview ↗

Response to Ndhk(1/3): Image-text relation sensitivity

Thank you for your careful reading and constructive questions. We appreciate your recognition of the consistent empirical gains and the matched-count synthesis/search comparisons. Your suggestions help strengthen the completeness and interpretability of our evaluation. We respond with targeted relation-sensitive diagnostics (Part 1), matched D3 and Direct Evolve comparisons (Part 2), and dimension-wise human assessment with raw-rating agreement (Part 3).

Q1 / Weakness 1: Direct behavioral evidence

Sections 4.2-4.3 and Appendix G examine effectiveness, transfer, and expression coverage. We extend these analyses with three complementary diagnostics. All diagnostic subsets are selected from the official FHM test-seen and test-unseen splits, with detector checkpoints and decision thresholds held fixed.

Confounder pairs and shared-anchor triples. We identify 60 human-verified harmful anchors, each with two benign counterparts: one retains the caption while changing the image; the other retains the underlying image while changing the caption. This gives 60 image-change pairs, 60 text-change pairs, and 60 shared-anchor triples.

For these diagnostics, we remove embedded captions and supply them separately as text. Same-image pairs reuse the exact same cleaned image; same-caption pairs reuse the exact same caption. Pair accuracy requires both predictions to be correct. Triple accuracy requires the harmful anchor and both benign counterparts to be correct:

Acctriple=1N∑j=1N1[y^h,j=1∧y^i,j=0∧y^t,j=0]\mathrm{Acc}_{\mathrm{triple}}=\frac{1}{N}\sum_{j=1}^{N}\mathbf{1}[\hat y_{h,j}=1\land\hat y_{i,j}=0\land\hat y_{t,j}=0].

Here, N=60N=60; hh, ii, and tt denote the harmful anchor, image-changed counterpart, and text-changed counterpart. A deterministic classifier relying exclusively on either modality cannot classify an entire triple correctly, because one pair has identical input in that modality but opposite labels.

Joint-interpretation subset. We select 100 examples (50 harmful/50 benign) requiring combined image-text interpretation. Criteria are fixed before selection, and annotators are blinded to detector predictions. We evaluate original multimodal input, text only, caption-removed image only, and cleaned image plus separately supplied original text. Cleaned images are inspected for residual captions. The last condition reuses exactly the image-only inputs, controlling for caption-removal effects. All methods and training seeds use the same diagnostic sets and frozen detectors.

Results. DE denotes Direct Evolve; no-D3 removes D3 from both reward and acceptance; Full is Reconstruct-then-Evolve. All three match accepted augmentation counts and detector-training budgets. Results are mean ± sample standard deviation over three detector-training seeds (%).

DiagnosticDEno-D3Full
Image-change pairs: both correct (60)65.00 ± 1.6768.89 ± 2.5576.67 ± 1.67
Text-change pairs: both correct (60)69.44 ± 2.5573.89 ± 1.9280.00 ± 1.67
Shared-anchor triples: all correct (60)52.78 ± 2.5557.78 ± 2.5571.67 ± 1.67
Joint subset: original multimodal (100)74.00 ± 1.0077.00 ± 1.0083.33 ± 1.53
Joint subset: text only (100)64.00 ± 1.0065.00 ± 1.0065.33 ± 1.53
Joint subset: cleaned image only (100)58.67 ± 0.5859.67 ± 0.5860.00 ± 1.00
Joint subset: cleaned image + text (100)73.67 ± 0.5876.67 ± 0.5883.00 ± 1.00

Relative to no-D3, Full improves image-change and text-change pair accuracy by 7.78 and 6.11 percentage points, and triple accuracy by 13.89 points. Relative to DE, the gains are 11.67, 10.56, and 18.89 points. Triple accuracy confirms that improved responses to both modality changes occur for the same anchors.

On the joint subset, Full improves multimodal accuracy by 6.33 points over no-D3 and 9.33 over DE, versus approximately 0.33 and 1.33 points in either unimodal condition. The cleaned-image-plus-text control preserves both multimodal gains.

D3 evaluates claim-specific contributions from both modalities, including mutually reinforcing evidence. The pair/triple tests directly probe responses to label-changing modality substitutions, while the joint subset tests interpretation with both modalities available. Together, these results support improved image-text relation use as a contributor to the downstream gains, alongside broader expression coverage.

AuthorsOpenReview ↗

Response to Ndhk(2/3): D3 attribution and reconstruction/evolution coupling

Q2: Matched D3 ablations

Section 4.4 and Figure 6(b) evaluate verification dimensions by removing each from both search reward and candidate acceptance. We extend the D3 ablation to a 2×2 design, separating these two uses under matched training conditions.

Each condition uses 8,500 reconstructed base examples, 3,050 accepted harmful augmentations, and 3,050 paired benign examples: 14,600 training examples. Detector initialization, optimizer, learning-rate schedule, global batch size, and optimizer-update count are identical. With global batch size 32, three epochs, and the final partial batch retained, each run uses 1,371 optimizer updates. Results are mean ± sample standard deviation over three matched detector-training seeds, evaluated on the paper's test-seen split.

Reward and acceptance. Without D3, the reward is R12=(D1D2)1/2/5R_{12}=(D_1D_2)^{1/2}/5; with D3, it is R123=(D1D2D3)1/3/5R_{123}=(D_1D_2D_3)^{1/3}/5. Acceptance requires D1=D2=5D_1=D_2=5, with D3=5D_3=5 additionally required when D3 acceptance is enabled.

D3 in rewardD3 in acceptanceAccuracy (%)Macro-F1 (%)
NoNo83.20 ± 0.3083.07 ± 0.33
YesNo84.00 ± 0.3083.88 ± 0.33
NoYes84.60 ± 0.2684.52 ± 0.28
YesYes85.37 ± 0.3185.28 ± 0.34

This table reports three-seed statistics; Figure 6(b) reports the original point estimates. With D3 acceptance fixed, adding D3 to reward improves accuracy by 0.77 points. With D3 reward fixed, adding D3 to acceptance improves accuracy by 1.37 points. Both uses contribute to the final performance.

Fixed-candidate-pool comparison. Acceptance decisions also influence subsequent search scheduling. To examine final selection directly, we apply both acceptance rules to the same complete pre-acceptance candidate pool, including rejected candidates and their recorded verification scores, while matching the final accepted count. Accuracy is 84.10 ± 0.20% under D1=D2=5D_1=D_2=5 and 85.37 ± 0.31% when additionally requiring D3=5D_3=5. The 1.27-point improvement demonstrates the contribution of D3-based selection within a fixed generated pool.

Q2 / Weakness 2: Reconstruction before controlled evolution

Section 4.2 compares MCTS, random search, and beam search at matched accepted counts; Section 4.4 examines reconstruction alternatives. We further evaluate Direct Evolve, which applies the same verifier-guided MCTS directly to original memes. The comparison distinguishes the detector's base training examples from the roots used for search:

ConfigurationBase data / MCTS rootsTraining countAccuracy (%)Macro-F1 (%)
Vanilla SFTOriginal / none8,50074.8073.75
Reconstruct onlyReconstructed / none8,50081.5081.38
Direct EvolveOriginal / original14,60081.80 ± 0.3081.65 ± 0.34
Reconstruct-then-EvolveReconstructed / reconstructed14,60085.37 ± 0.3185.28 ± 0.34

The first two rows reproduce Table 1 point estimates; the augmented configurations report three-seed statistics on test-seen. Both add 3,050 harmful-benign pairs and share source-example IDs, semantic anchors, mutation operators, generator, verifier, full D3 configuration, maximum search budget per seed, and detector-training budget.

Reconstruct-then-Evolve improves accuracy by 3.57 points and macro-F1 by 3.63 points over Direct Evolve. This comparison measures the benefit of reconstruction across the complete training pipeline: both the base training examples and the search roots are reconstructed before controlled expansion.

Part 1 connects this pipeline-level advantage to relation-sensitive behavior. Compared with Direct Evolve, the full method gains 11.67 points on image-change pairs, 10.56 points on text-change pairs, and 18.89 points on shared-anchor triples. Thus, reconstruction before evolution improves both overall detection and responses to controlled changes in image-text relationships.

Together, these comparisons identify the contributions of D3-guided search, D3-based final selection, and the complete reconstruction-then-evolution workflow under matched accepted counts and detector-training budgets.

AuthorsOpenReview ↗

Response to Ndhk(3/3): Dimension-wise human evaluation and agreement

Q3 / Weakness 1: Independent D1-D3 assessment

Appendix B and Table B.1 report semantic preservation for reconstruction and overall validity for accepted evolution samples. We extend this assessment with dimension-specific ratings, failure categories, and agreement computed from individual ratings before aggregation.

Protocol. We assess the same 160 accepted evolved samples. Three human assessors independently assign 1-5 scores for D1 (realization quality), D2 (harmful-chain completeness and fidelity), and D3 (cross-modal contribution), producing 1,440 individual ratings. They inspect the final rendered memes and receive the source semantic anchor for D2 judgments. They are blinded to automatic verifier scores and the earlier overall assessment.

We also obtain independent image-input assessments from GPT-6.1 Terra (gpt-6.1-terra) and Gemini 3.8 Flash (gemini-3.8-flash). Each uses the same rubric, with a separate request for every sample-dimension combination: 480 ratings per model. Neither evaluator participates in generation or acceptance.

Dimension-wise validity. Human validity follows Appendix B: average the three ratings for each sample and dimension, round to an integer, and require at least 4. Each API score must be at least 4. Joint validity requires the same sample to pass all three dimensions. Entries below give mean score / validity % (valid count). Human mean scores use all 480 raw ratings per dimension, before sample-level rounding.

DimensionHuman assessorsGPT-6.1 TerraGemini 3.8 Flash
D14.78 / 97.50 (156/160)4.80 / 98.13 (157/160)4.74 / 96.88 (155/160)
D24.67 / 95.63 (153/160)4.69 / 96.25 (154/160)4.61 / 94.38 (151/160)
D34.58 / 93.75 (150/160)4.60 / 94.38 (151/160)4.49 / 91.88 (147/160)
All three valid91.25 (146/160)92.50 (148/160)89.38 (143/160)

D3 has the lowest validity across the human and both external evaluations, identifying claim-specific cross-modal contribution as the most challenging dimension.

Ordinal agreement and failures. We compute ordinal Krippendorff's α\alpha separately for each dimension from its original 3×160 human-rating matrix, before averaging, rounding, or adjudication. The 95% confidence intervals use 4,000 item-level bootstrap resamples, retaining all three ratings for each sampled item.

DimensionOrdinal alpha95% bootstrap CIPrimary failure categories
D10.844[0.750, 0.923]Visual artifacts: 3; layout/readability problems: 1
D20.809[0.721, 0.887]Target/proposition drift: 4; incomplete/unsupported harmful chain: 3
D30.784[0.699, 0.858]Generic visuals: 5; weak claim-specific image-text support: 3; decorative imagery: 2

Each failed sample-dimension receives one primary category under the rounded-mean criterion. The 4 D1, 7 D2, and 10 D3 failures total 21 dimension-level failures across 14 distinct samples, consistent with 146/160 samples passing all three dimensions.

Sensitivity to rounding. Requiring the unrounded mean human score to be at least 4 gives validity rates of 97.50%, 95.00%, and 92.50% for D1, D2, and D3, respectively, with 143/160 (89.38%) jointly valid. Rounding changes joint validity for three samples. Ordinal alpha is unchanged because it uses individual ratings directly.

Connection to Appendix B. The original overall-score distribution for levels 1-5 remains (0, 2, 12, 32, 114), yielding a mean of 738/160 = 4.6125, reported as 4.61, and overall validity of 146/160 = 91.25%. This overall assessment and the new joint-dimensional assessment are separate measures with matching valid counts. The new audit adds dimension-level resolution and agreement before aggregation.

Together with the relation-sensitive diagnostics in Part 1 and matched interventions in Part 2, these findings connect synthesis decisions, generated-example quality, and downstream detector behavior.

Thank you again for these thoughtful suggestions, which have helped us strengthen the paper's empirical support and clarify its interpretation. We welcome further questions or recommendations and would be glad to address any additional points. We will incorporate all additional results and analyses presented here into the final manuscript and appendix, together with detailed descriptions of subset provenance, image preprocessing, training controls, human and API evaluation, and statistical procedures.

Reviewer_wAtdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Existing methods primarily focus on model architectures and reasoning capabilities while overlooking limitations in the training data, particularly ambiguous cross-modal semantics and insufficient diversity in harmful expressions. To address these limitations, the proposed framework consists of two stages. First, the Reconstruct stage decomposes harmful memes into their underlying semantics and surface realizations, generating semantically grounded and more explicit training samples. Second, the Evolve stage employs verifier-guided Monte Carlo Tree Search (MCTS) to diversify the reconstructed memes through multiple mutation strategies while attempting to preserve their harmful semantics. The framework further generates corresponding harmless samples and uses the synthesized data for supervised fine-tuning. Experiments on FHM, MAMI, HarMeme, and PrideMM report consistent improvements over selected baselines and conventional fine-tuning settings.

Critical strengths

1.Relevant and meaningful research problem: The paper addresses an important challenge in harmful meme detection by shifting attention from model-centric improvements to training data quality and diversity. This data-centric perspective is meaningful because harmful semantics often depend on subtle interactions between visual and textual information.

2.Promising empirical improvements across multiple benchmarks. The authors evaluate the proposed framework on four harmful meme detection benchmarks and report consistent improvements over conventional fine-tuning. The evaluation also includes cross-backbone experiments and ablation studies, providing preliminary evidence that the synthesized data may benefit different detection models.

Critical weaknesses
  1. The central motivation and the effectiveness of semantic reconstruction are not sufficiently justified. The paper identifies the implicit and ambiguous nature of harmful memes as a major challenge. However, the Reconstruct stage addresses this difficulty by transforming original memes into more explicit and semantically grounded examples.This raises an important conceptual question: if the original challenge lies in recognizing subtle and implicit harmful semantics, why should training on simplified or more explicit reconstructions improve the model's ability to interpret such ambiguity?Although the reported benchmark improvements provide empirical support for the approach, they do not fully explain the underlying mechanism. In particular, the paper lacks a sufficiently targeted evaluation of whether the method improves reasoning over genuinely ambiguous or implicit image-text interactions, rather than primarily benefiting from easier or more explicit training examples.

  2. The performance improvements are not adequately disentangled from the effects of increased training data volume. In the Evolve stage, the framework introduces additional synthesized harmful memes and corresponding harmless samples into the training set. Consequently, the final training distribution changes in both sample quantity and diversity. Although the paper includes stage-wise ablations, these comparisons do not fully isolate the contribution of the proposed evolution strategy from the general benefits of additional synthetic data.More controlled comparisons are needed, such as random augmentation, conventional image generation, or simpler diversity-based sampling under matched training-set sizes and computational budgets. Without such controls, it remains unclear whether the reported improvements primarily arise from the proposed MCTS-guided evolution mechanism or simply from the increased amount of training data.

  3. The experimental comparisons and reported baseline results require further clarification. The main comparison includes methods based on different architectures and potentially different training configurations. It is not sufficiently clear whether all reported results are obtained under comparable data splits, preprocessing procedures, and evaluation settings. In addition, some baseline entries warrant careful verification against their original publications. The authors should explicitly distinguish reproduced results from numbers taken from previous papers, provide the exact sources of the reported values, and clarify the comparability of the evaluation protocols. These details are important for assessing whether the claimed improvements over prior methods constitute a reliable and fair comparison.

  4. The necessity and efficiency of the MCTS-based evolution strategy are not sufficiently established. The Evolve stage adopts Monte Carlo Tree Search to explore multiple mutation strategies and select high-quality synthetic samples. However, it is not clear whether the performance gains depend on the tree-search mechanism itself or can be achieved through simpler candidate generation and selection strategies. Although the paper studies the effects of individual mutation operators and verification dimensions, a controlled comparison against random search, greedy selection, or other simpler approaches under the same candidate-generation budget would provide stronger evidence for the necessity of MCTS.

  5. Some methodological explanations and experimental analyses lack sufficient depth. The overall framework is understandable, but several methodological components are introduced primarily at a procedural level, without sufficiently explaining their specific relevance to harmful meme understanding. In particular, the relationship between semantic decomposition, reconstruction quality, and the preservation of implicit harmful meaning deserves more detailed analysis. Similarly, some experimental discussions mainly describe numerical improvements without adequately explaining their causes or limitations. A more focused analysis of failure cases and performance differences across datasets would strengthen the paper.

Questions for authors

Q1. Since the Reconstruct stage converts ambiguous harmful memes into more explicit examples, could the authors explain why this process is expected to improve the detection of genuinely implicit harmful semantics? Could the authors provide a controlled evaluation on particularly ambiguous or reasoning-intensive test samples?

Q2. Could the authors compare the proposed Evolve stage against simpler data augmentation methods under matched training-set sizes and training budgets? In particular, how much improvement can be attributed specifically to MCTS-guided evolution rather than increasing the number of synthetic samples?

Q3. How do the authors verify that the harmful semantics and labels remain consistent throughout reconstruction and evolution? Is there any human evaluation of semantic preservation, label correctness, or the quality of generated harmless counterparts?

Q4. Could the authors clarify which baseline results are reproduced and which are taken from previous publications? Are all comparisons conducted using consistent dataset splits and evaluation protocols? Please also verify the reported baseline values against their original sources.

Q5. What advantages does MCTS provide over random search or greedy candidate selection under an equivalent generation and verification budget? Could the authors provide a controlled comparison and report the associated computational overhead?

Q6. How does the synthesized training data affect performance on different categories of harmful memes, particularly those requiring implicit reasoning, cultural context, or nontrivial image-text interactions? Could the authors provide a more detailed failure-case analysis?

3 comments

Dear Reviewer #wAtd,

We would like to kindly follow up on our rebuttal. We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns or if any issues remain unresolved. Thank you for your time and consideration!

Thank you for the detailed rebuttal. The newly added experimental results address my earlier concerns regarding the research motivation and performance attribution. Nevertheless, the overall novelty of the method remains limited, as the approach primarily assembles and integrates existing techniques. I also retain doubts about the necessity of this two-stage framework. In my view, the authors have not yet sufficiently clarified the unique methodological contributions of this two-stage design compared with conventional data reconstruction and augmentation pipelines.

Overall, I acknowledge the efforts made by the authors. However, I still have reservations about the contributions of this method, and therefore maintain my original score.

Dear Reviewer wAtd,

Thank you for acknowledging that our experiments address your concerns about motivation and performance attribution! We respectfully disagree that our method “primarily assembles and integrates existing techniques.” Reconstruct introduces two linked design choices whose value can be examined before Evolve adds a single example.

Q1. What does Reconstruct contribute beyond ordinary data augmentation?

First, we decouple the source semantics from their realization (§3.1, Eq. 1). The semantic anchor records the target, attributed property, harm category, and core proposition. The realization records how the image and caption convey that proposition through a particular relation, rhetorical strategy, scene, or template. In an ironic comparison, for example, the target and the claim about it belong to the anchor. The visual juxtaposition that conveys the irony belongs to the realization. An edit can preserve the label harmful while changing the target or claim. The anchor makes that change detectable.

Second, this decomposition defines the scope of a minimal repair (§3.2). With the source proposition and caption fixed, we ask whether the image supplies the entity, relation, or contrast that the caption presupposes. A grounded source receives no semantic repair. If one visual link is missing and can be restored faithfully, only that link may change. The original specification states, “Repair exactly the named link, nothing else” (original Appendix J.1, now K.1). The anchor specifies what must be preserved. The diagnosis specifies whether and where the realization may be edited.

To our knowledge, this is the first work in harmful meme detection to use an explicit decomposition of semantics and realization to diagnose and minimally repair a missing visual premise while preserving the source proposition and caption. Kiela et al. (2020, §2.2) reconstructed memes using meaning-preserving replacement photographs. HateSieve (Su et al., 2025) generates related memes for contrastive training. Our method decides which part of a source realization needs repair, if any, and carries the recovered proposition forward as the reference for subsequent edits.

Q2. Does Reconstruct help before the training set grows?

With 8,500 examples in both conditions, replacing the originals one for one with reconstructions raises FHM test-seen accuracy from 74.80% to 81.50% (§4.5, Table 2). The original Figure 6(a) compares Naive Generation (77.20%), Caption Distillation (78.10%), Ungrounded Reconstruction (79.60%), and Grounded Reconstruct (81.50%). The closest alternative already reproduces the source scene, composition, and caption meaning. Grounded Reconstruct gains 1.90 points over it.

The rebuttal holds training captions identical: replacing generic imagery with claim-specific evidence raises correct predictions from 74 to 79 on 100 unchanged memes requiring joint interpretation. Reconstruct with the original captions also reaches 79/100 (Appendix J.2, Table J.4).

Q3. Why reconstruct before evolution, and why use Evolve on the reconstructed data?

Evolve varies the realization and checks each rendered descendant against the same source proposition (D2), while verifying that both modalities contribute to the interpretation (D3). On FHM test-seen, Direct Evolve reaches 81.80% and the full method 85.37% with matched source IDs, operators, generator, verifier, accepted pair counts, and training updates (§4.5, Table 2, three-seed means). This measures the joint benefit of reconstructing the training base and search roots.

On the same reconstructed base, one-shot synthesis reaches 82.30%, diversity selection 83.10%, and Evolve 85.30% at matched training exposure (Appendix J.3, Table J.5, one checkpoint per condition). The original Figure G.1 shows what gains coverage: metaphorical and symbolic expressions rise from 5.4% to 9.1% on FHM and from 3.2% to 7.0% on MAMI.

Your initial review recognized our “data-centric perspective” as “meaningful” and asked for controls on implicit interpretation, training volume, and search. We appreciate that recognition and conducted those comparisons. This perspective distinguishes our contribution from work on detector architecture or inference: the object we diagnose, repair, and diversify is the image–text evidence in the training example itself. The broader objection that our method assembles existing techniques was first stated explicitly in your follow-up. We would welcome a comparison with the prior method you consider closest on these specific operations. In light of the original design and the evidence addressing your questions, we respectfully ask you to reconsider your assessment of the contribution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewer wAtd (2/3): Augmentation and search efficiency

We address Weaknesses 2/4 and Questions 2/5 with a controlled comparison that matches training exposure and caps generation/verification calls. MCTS retains gains over repetition, independent synthesis, diversity selection, and simpler search policies under these controls.

Controlling data quantity and training exposure.

Every condition contains 8,500 reconstructed base examples plus 3,050 harmful and 3,050 benign additions: 14,600 training entries. Repetition copies base examples to match these class counts. All conditions share initialization, optimizer, learning-rate schedule, three epochs, learning rate 5e-5, and effective batch size 32. Retaining each epoch's final partial batch gives 3 × ceil(14,600/32) = 1,371 optimizer updates.

Augmentation ruleAccuracy (%)Macro-F1 (%)
Class-matched repetition82.1081.90
Independent one-shot synthesis82.3082.12
Diversity selection83.1082.97
Uniform-node random search82.0081.80
Greedy search83.0082.86
Beam search (width 64)83.1082.98
MCTS Evolve85.3085.22

Evaluation uses the same 1,000-example FHM test-seen manifest and one prespecified detector checkpoint per condition. Independent synthesis generates directly from seed specifications. Diversity selection uses fixed embeddings and farthest-point sampling from the same qualifying one-shot pool used by its random-selection comparator.

All synthesis conditions receive the call caps below. After common duplicate filtering, 3,050 harmful examples are selected from each sufficient pool by seeded uniform sampling without replacement; only the diversity condition changes this selection rule. Each selected example receives an accepted benign counterpart.

MCTS exceeds repetition, one-shot synthesis, and diversity selection by 3.20, 3.00, and 2.20 accuracy points. Its advantages over random, greedy, and beam search are 3.30, 2.30, and 2.20 points. The improvement persists with data quantity, class counts, and training updates fixed.

Candidate yield under matched call budgets.

Random search samples available parents uniformly; greedy expands the highest-reward parent with fixed tie-breaking/restarts; beam retains 64 candidates; MCTS uses UCT. All share semantic seeds, four mutation operators, cluster scheduling, generator, verifier, and acceptance rules.

Each policy has caps of 25,091 planning, 25,091 rendering, and 75,273 dimension-specific verification calls, with shared generation settings and token limits. Failed calls and retries consume these caps. The budget is defined in API calls; completed proposals and actual calls are recorded separately. Each policy completes 25,091 proposals.

PolicyQualifyingYield (%)Search hController h
Random4,25616.9612.340.01
Greedy4,96819.8012.400.05
Beam-645,34721.3112.490.16
MCTS6,12224.4012.520.21

Qualifying counts precede duplicate filtering; yield is qualifying/completed proposals. Both tables describe the same controlled study: each search pool supplies 3,050 distinct harmful examples for the size-matched detector comparison above.

MCTS produces 1,866 more qualifying candidates than random search and 775 more than beam, relative increases of 43.84% and 14.49%. Its higher yield accompanies greater downstream utility after matching the selected training-set size.

Computational accounting and relation to the original results.

Timing covers harmful-candidate planning, rendering, verification, and controller execution on the same eight-H800 allocation. Controller time is included in search time. MCTS takes 0.18 h more than random search (+1.46%); its controller occupies 0.21 h, or 1.68% of search time. Benign construction is recorded separately by proposal attempts, accepted counts, and runtime.

App. H's 65.5 aggregate GPU-hours is an arithmetic estimate for the original 25,091-candidate trajectory, assuming 989 TFLOP/s per GPU and 50% MFU. It has a different accounting basis from measured wall time and excludes reconstruction, seed preparation, benign construction, detector training/evaluation, and system overhead.

The present full-pool selection protocol differs from the generation-order-prefix protocol in Sec. 4.2. Its 85.30/85.22 accuracy/macro-F1 is a single-checkpoint result. At FHM 1.0x, the random/beam three-seed mean accuracies reported in Sec. 4.2 are 82.00/82.80%; the MCTS results used here are 85.37 ± 0.31% accuracy and 85.28 ± 0.34% macro-F1 (three-seed mean ± sample standard deviation). Table 1 reports the original full-pipeline result of 85.50/85.45. These results retain their respective selection and aggregation protocols.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.