acceptodds
Under review as a conference paper at ICLR 2027

HOLOGRAM: A Graph-Conditioned Context Steering Framework for Multilingual, Multimodal, and Multi-Granular Toxic Content Detection

16 views

Abstract

In this work, we introduce a novel graph-conditioned steering framework HOLOGRAM, which is among the first content moderating unified architecture that adapts a frozen VLM to these three axes of moderation heterogeneity through the same mechanism. For each post, HOLOGRAM constructs a semantic moderation neighborhood from training. Subsequently, a graph-based-encoder models how nearby examples support, duplicate, or contrast with one another. Its representations select representative artifacts to generate a query-conditioned low-rank context vector inside every frozen decoder layer of the VLMs. The resulting framework supports multiple modalities, languages, and supports labels at both, binary and fine-grained levels. We evaluate our framework across an array of established vector (and memory)-steering methods and prove that our graph-conditioned steering framework not only outperforms prior methods, but also yields high consistency. The results demonstrate the potential of local relational residual context addition as a general interface for heterogeneous social-media moderation and proves its strong usability, even towards production deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
hateful content detection, multimodal, multilingual, hate speech, hate memes

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_8ehUOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

HOLOGRAM is a graph-conditioned activation-steering method for toxic-content classification across languages, modalities, and label granularities. For each query, it builds a neighborhood of training examples, applies a GATv2 encoder, selects representative supports, and converts them into query-conditioned low-rank context vectors injected into every frozen VLM decoder layer. The paper evaluates Qwen3-VL and Intern3.5-VL at 2B and 8B scales on HateXplain, Amharic, HASOC, BHM, and ToxicTags in binary and fine-grained settings.

Critical strengths
  1. The method gives a coherent interface for heterogeneous moderation tasks. Query-conditioned graph support and per-layer low-rank injection are a technically combination: the VLM remains frozen, while the retrieved evidence can influence different decoder layers without being appended as a long prompt.
  2. The experiments cover two VLM families, two model sizes, multilingual text, multimodal memes, and binary versus fine-grained labels. The comparisons with both prompt-based methods and activation-steering baselines, together with the combined-suite and random-context analyses.
Critical weaknesses
  1. Appendix D.4 replaces support representations with Gaussian vectors only at inference in an already trained model. This demonstrates sensitivity to the learned input distribution, but does not establish the benefit of graph structure. Since the query itself enters the graph, a trained query-only encoder or a support-set encoder without edges could explain part of the gain. Neither control is reported.
  2. The baseline comparisons change support selection and adaptation together. ICL uses similarity-ranked examples, whereas HOLOGRAM adds clustering, diversity selection, contrastive supervision, and layer-wise adaptation. Their individual contributions remain unclear. Training for three rather than five epochs also does not establish lower cost: each HOLOGRAM training query requires separate VLM passes for three shot counts (Section 4.5).
Questions for authors
  1. How does HOLOGRAM compare with trained query-only and permutation-invariant support-set encoders using the same context builder and layer-wise steering, with comparable supervision and parameter budgets?
  2. Do the gains persist when ICL uses the same support-selection policy and PMT/LIVE receive comparable parameter and training-compute budgets? Please report trainable parameter counts, VLM training passes, and measured training and inference costs.
10 comments
AuthorsOpenReview ↗

Rebuttal: Part 1

We thank the reviewer for very insightful and constructive reviews with multiple feedbacks that have significantly bolstered the robustness of our paper. Based on the extensive ablations and other experiments we have carried out, we are now even more confident in the real world adaptation of HoloGram.

Query 1: Missing ablations

Response 1: We sincerely thank you that you took your time to go through the ablation which we have done in the Appendix. Based on your and other reviewers’ feedback, we ran extensive ablations to demonstrate that HoloGram as a whole is powered through GNNs as well as through the layer-specific memory steers. We present the results and corresponding observations/takeaways one by one below:

All results below report Macro-F1 Score across 2-shot, 4-shot, and 8-shot settings (k ∈ {2, 4, 8}) on Toxic Tags (TT) and HASOC (Hindi). The model is Qwen-VL with 2B parameters. Every ablation varies a single component at a time around our unified default configuration (Ours: knn = 6, λ = 0.8, L_CE + L_SCL, C = 10, r = 64, multi-layer memory injection). Bold = best at our setting; italics = ours when second-best.


1. Architecture, GNN & Baseline Ablations

VariantTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
(1) Permutation-invariant set encoder → MLP49.3155.5955.1955.7557.6254.31
(2) Top-k raw candidate vectors → mean → MLP53.2552.6053.3155.1253.5755.36
(3) Memory injected at last decoder layer only32.9236.8125.0638.9840.5431.59
(4) PMT (rank-64 M_l = A_l B_l per layer; no retrieval)*—53.20——44.46—
(5) LoRA adapter—52.12——43.43—
(6) Ours (Full GNN + multi-layer memory)52.2356.8856.1654.2559.4357.34

Note that PMT does not support retrieval or few-shot conditioning (no k); its single shot-independent score is placed in the central 4-shot column for direct comparison. Also, LoRA rank = 64; shot-independent baseline placed in the central 4-shot column.

Here are the key takeaways:

  • Graph relational modeling matters over pooling/sets: At our primary 4-shot setting, our GNN outperforms the permutation-invariant set encoder by +1.29 on TT (56.88 vs. 55.59) and +1.81 on Hindi (59.43 vs. 57.62), and outperforms mean-pooling + MLP by +4.28 on TT and +5.86 on Hindi (mean-pooling plateaus across shot counts rather than exploiting additional candidates).
  • Selective-layer memory injection does not perform better: Restricting memory injection to only the last decoder layer causes severe degradation across all shot counts (-20.07 on TT and -18.89 on Hindi at 4 shots).
  • Substantial gains over non-retrieval PEFT/memory baselines: Even when PMT uses per-layer rank-64 memory (M_l = A_l B_l), our retrieval-conditioned GNN achieves +3.68 over PMT and +4.76 over LoRA on TT, and +14.97 over PMT / +16.00 over LoRA on Hindi at 4 shots.

2. Training Objective (Loss) Ablation

Loss ConfigurationTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
loss_ce_only52.6654.9652.0956.3258.8254.64
loss_scl_only27.2127.2127.2135.4435.4435.44
loss_ce_and_scl (Ours)52.2356.8856.1654.2559.4357.34

Following are the inferred observations:

  • loss_scl_only lacks token/task supervision and collapses to a constant baseline (27.21 on TT, 35.44 on Hindi) regardless of shot count.
  • Combining L_CE + L_SCL consistently improves 4-shot performance (+1.92 on TT, +0.61 on Hindi) and prevents degradation when scaling to 8 shots (+4.07 on TT, +2.70 on Hindi over CE alone).
AuthorsOpenReview ↗

Rebuttal: Part 2

3. Graph & Retrieval Hyperparameter Sensitivity (knn, C, λ)

3a. Graph Edge Connectivity (knn)

knn EdgesTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
knn = 351.0155.4352.2454.3555.2353.04
knn = 450.2355.1656.4554.6659.1857.30
knn = 551.7355.5556.7452.2255.6551.05
knn = 6 (Ours)52.2356.8856.1654.2559.4357.34

3b. Number of Graph Nodes (C)

Nodes (C)TT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
C = 551.5953.3353.9253.7754.9953.87
C = 10 (Ours)52.2356.8856.1654.2559.4357.34
C = 2054.2957.4457.6154.8258.5352.52

3c. MMR Diversity Weight (λ)

MMR λTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
λ = 0.2549.7454.0453.8153.5860.8357.61
λ = 0.550.3755.2454.8556.1658.8755.26
λ = 0.8 (Ours)52.2356.8856.1654.2559.4357.34

Key takeaways are as follows:

  • knn = 6 delivers the highest 4-shot score on both datasets (56.88 TT, 59.43 Hindi).
  • C = 10 is the optimal cross-dataset sweet spot: moving from C = 5 to C = 10 yields major gains at 4 shots (+3.55 TT, +4.44 Hindi), whereas C = 20 doubles graph size for only +0.56 on TT while hurting Hindi (-0.90 at 4 shots, -4.82 at 8 shots).
  • λ = 0.8 provides strong cross-dataset robustness without per-dataset tuning: it improves TT monotonically across 2, 4, and 8 shots (+2.84 over λ = 0.25 at 4 shots) while remaining competitive (second-best) on Hindi.

4. Memory Capacity (Rank r) Ablation

Memory Rank (r)TT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
r = 846.9051.4152.7549.6654.5250.85
r = 1648.9052.8152.6551.4558.0955.32
r = 3249.9755.0656.1550.6359.1955.76
r = 64 (Ours)52.2356.8856.1654.2559.4357.34

Key Takeaways:

  • Performance scales monotonically with memory rank r at our primary 4-shot setting on both benchmarks (51.41 → 52.81 → 55.06 → 56.88 on TT, +5.47; 54.52 → 58.09 → 59.19 → 59.43 on Hindi, +4.91), confirming that higher-rank per-layer capacity (r = 64) is effectively utilized. These ablations further strengthen the robustness and usability of our framework.
AuthorsOpenReview ↗

Rebuttal: Part 3

Query 2: Parameters and inference efficiency

Response 2: First of all, we thank you for pointing this out. Below we present the trainable parameter efficiency of our framework:

ModelGNN (%)Learnable Memory (%)Total (%)
Qwen3-VL 2B0.444%0.487%0.931%
InternVL3.5 2B0.402%0.441%0.843%
Qwen3-VL 8B0.108%0.277%0.384%
InternVL3.5 8B0.111%0.284%0.395%

As we can observe, HoloGram adds less than 1% total overhead (0.843%–0.931% at 2B and 0.384%–0.395% at 8B). Also, moving from 2B to 8B more than halves the relative total overhead (0.931% → 0.384% for Qwen3-VL; 0.843% → 0.395% for InternVL3.5), with the GNN share dropping nearly 4× (~0.40–0.44% → ~0.11%). This further shows the efficiency of trainable parameters. Most importantly, for all setups, the model is loaded only once and PyTorch’s Hook is applied for inference-time steering using GNN and learnable memory parameters for different setups, which tunnel down to a highly efficient system. Finally, we also present inference stats for Qwen - VL 2B model:

DatasetMethodkPrompt TokensGenerated TokensGeneration (ms)Peak GPU Memory (GiB)
Hindi (T, 4)Zero-shot03642.984.38.15
ICL (cosine top-k)81,5843.0142.98.47
HoloGram83642.6100.28.81
ToxicTags (M, 4)Zero-shot06712.5115.98.37
ICL (cosine top-k)83,8992.9467.09.33
HoloGram86712.7150.710.05

Because HoloGram conditions via latent memory rather than stuffing retrieved examples into the context window, it keeps prompt length identical to zero-shot; reducing prompt tokens over 8-shot ICL by 4.35× on Hindi (364 vs. 1,584, -77.0%) and 5.81× on multimodal ToxicTags (671 vs. 3,899, -82.8%). Further, it is up to 3.10× faster inference than ICL:. On multimodal ToxicTags (k = 8), HoloGram cuts generation latency from 467.0 ms to 150.7 ms (3.10× faster, -67.7%), and on text Hindi from 142.9 ms to 100.2 ms (1.43× faster, -29.9%), adding only 15.9–34.8 ms over zero-shot. This leads only to a modest memory footprint. Peak GPU memory increases by only +0.34 GiB on Hindi (8.81 vs. 8.47 GiB) and +0.72 GiB on ToxicTags (10.05 vs. 9.33 GiB) relative to 8-shot ICL.

Reviewer_QAybOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies toxic-content detection across languages, modalities, and label granularities. HOLOGRAM retrieves training examples, builds a query-centered graph, and converts its representations into low-rank memories for each layer of a frozen vision-language model. Classification and supervised contrastive losses are used to train the external modules. Experiments cover five datasets, four backbones, and additional binary mixtures, with improved Macro-F1 over prompting, PMT, and LIVE.

Critical strengths
  1. Query-dependent retrieval combined with layer-wise steering is a reasonable design, and Section 4 describes the mechanism concretely.

  2. Table 2 covers 36 model/dataset configurations, including Amharic, Hindi, and Bengali memes, with substantial gains across both model families.

  3. Appendix D includes random-context controls, representation analyses, and per-source results for the combined suites. It also acknowledges several limitations of these analyses.

Critical weaknesses
  1. The central architectural contribution is not isolated. Appendix D.4 replaces genuine representations with random vectors only at inference. This shows that the trained network relies on its inputs, but does not distinguish graph structure from simpler supervised adaptation. A trained no-graph aggregator and static layer-wise memory are important controls. A comparable LoRA baseline can also clarify whether the extra retrieval and graph machinery is necessary.

  2. The evidence supports a shared architecture more clearly than a single unified model. Appendix C confirms that Table 2 trains each dataset separately. Its joint experiments use two separate binary suites, with no single model covering modalities and label granularities together. Qwen's BHM result also falls from 67.04 separately to 59.93 jointly at eight shots (Tables 2 and 7). Training sizes vary, so this needs to be explained before attributing the decrease to joint training.

  3. The efficiency claims lack measurements. Constant prompt length does not imply constant inference cost, because retrieval, graph construction, and the embedding model add computation (Sections 4.1–4.2). Three training epochs are not directly comparable with five baseline epochs because Section 4.5 requires three VLM passes per query. Parameter counts, latency, and training costs are needed.

  4. The final evaluation protocol needs clarification. Table 6 attributes differences from Table 2 to a corrected answer parser and run-to-run variation. It remains unclear whether all main baselines use the corrected parser. Section 5 also leaves validation-based parameter selection and training-seed variability unspecified. These reporting issues are fixable, but affect confidence in the margins.

Questions for authors

As shown in the weaknesses.

10 comments
AuthorsOpenReview ↗

Rebuttal: Part 1

We thank the reviewer for very insightful and constructive reviews with multiple feedbacks that have significantly bolstered the robustness of our paper. Based on the extensive ablations and other experiments we have carried out, we are now even more confident in the real world adaptation of HoloGram.

Query 1: Missing ablations

Response 1: We sincerely thank you that you took your time to go through the ablation which we have done in the Appendix. Based on your and other reviewers’ feedback, we ran extensive ablations to demonstrate that HoloGram as a whole is powered through GNNs as well as through the layer-specific memory steers. We present the results and corresponding observations/takeaways one by one below. We also provide the results of the statistical significance test as a response to the 4th query.

All results below report Macro-F1 Score across 2-shot, 4-shot, and 8-shot settings (k ∈ {2, 4, 8}) on Toxic Tags (TT) and HASOC (Hindi). The model is Qwen-VL with 2B parameters. Every ablation varies a single component at a time around our unified default configuration (Ours: knn = 6, λ = 0.8, L_CE + L_SCL, C = 10, r = 64, multi-layer memory injection). Bold = best at our setting; italics = ours when second-best.


1. Architecture, GNN & Baseline Ablations

VariantTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
(1) Permutation-invariant set encoder → MLP49.3155.5955.1955.7557.6254.31
(2) Top-k raw candidate vectors → mean → MLP53.2552.6053.3155.1253.5755.36
(3) Memory injected at last decoder layer only32.9236.8125.0638.9840.5431.59
(4) PMT (rank-64 M_l = A_l B_l per layer; no retrieval)*—53.20——44.46—
(5) LoRA adapter—52.12——43.43—
(6) Ours (Full GNN + multi-layer memory)52.2356.8856.1654.2559.4357.34

Note that PMT does not support retrieval or few-shot conditioning (no k); its single shot-independent score is placed in the central 4-shot column for direct comparison. Also, LoRA rank = 64; shot-independent baseline placed in the central 4-shot column.

Here are the key takeaways:

  • Graph relational modeling matters over pooling/sets: At our primary 4-shot setting, our GNN outperforms the permutation-invariant set encoder by +1.29 on TT (56.88 vs. 55.59) and +1.81 on Hindi (59.43 vs. 57.62), and outperforms mean-pooling + MLP by +4.28 on TT and +5.86 on Hindi (mean-pooling plateaus across shot counts rather than exploiting additional candidates).
  • Selective-layer memory injection does not perform better: Restricting memory injection to only the last decoder layer causes severe degradation across all shot counts (-20.07 on TT and -18.89 on Hindi at 4 shots).
  • Substantial gains over non-retrieval PEFT/memory baselines: Even when PMT uses per-layer rank-64 memory (M_l = A_l B_l), our retrieval-conditioned GNN achieves +3.68 over PMT and +4.76 over LoRA on TT, and +14.97 over PMT / +16.00 over LoRA on Hindi at 4 shots.

2. Training Objective (Loss) Ablation

Loss ConfigurationTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
loss_ce_only52.6654.9652.0956.3258.8254.64
loss_scl_only27.2127.2127.2135.4435.4435.44
loss_ce_and_scl (Ours)52.2356.8856.1654.2559.4357.34

Following are the inferred observations:

  • loss_scl_only lacks token/task supervision and collapses to a constant baseline (27.21 on TT, 35.44 on Hindi) regardless of shot count.
  • Combining L_CE + L_SCL consistently improves 4-shot performance (+1.92 on TT, +0.61 on Hindi) and prevents degradation when scaling to 8 shots (+4.07 on TT, +2.70 on Hindi over CE alone).
AuthorsOpenReview ↗

Rebuttal: Part 2

3. Graph & Retrieval Hyperparameter Sensitivity (knn, C, λ)

3a. Graph Edge Connectivity (knn)

knn EdgesTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
knn = 351.0155.4352.2454.3555.2353.04
knn = 450.2355.1656.4554.6659.1857.30
knn = 551.7355.5556.7452.2255.6551.05
knn = 6 (Ours)52.2356.8856.1654.2559.4357.34

3b. Number of Graph Nodes (C)

Nodes (C)TT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
C = 551.5953.3353.9253.7754.9953.87
C = 10 (Ours)52.2356.8856.1654.2559.4357.34
C = 2054.2957.4457.6154.8258.5352.52

3c. MMR Diversity Weight (λ)

MMR λTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
λ = 0.2549.7454.0453.8153.5860.8357.61
λ = 0.550.3755.2454.8556.1658.8755.26
λ = 0.8 (Ours)52.2356.8856.1654.2559.4357.34

Key takeaways are as follows:

  • knn = 6 delivers the highest 4-shot score on both datasets (56.88 TT, 59.43 Hindi).
  • C = 10 is the optimal cross-dataset sweet spot: moving from C = 5 to C = 10 yields major gains at 4 shots (+3.55 TT, +4.44 Hindi), whereas C = 20 doubles graph size for only +0.56 on TT while hurting Hindi (-0.90 at 4 shots, -4.82 at 8 shots).
  • λ = 0.8 provides strong cross-dataset robustness without per-dataset tuning: it improves TT monotonically across 2, 4, and 8 shots (+2.84 over λ = 0.25 at 4 shots) while remaining competitive (second-best) on Hindi.

4. Memory Capacity (Rank r) Ablation

Memory Rank (r)TT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
r = 846.9051.4152.7549.6654.5250.85
r = 1648.9052.8152.6551.4558.0955.32
r = 3249.9755.0656.1550.6359.1955.76
r = 64 (Ours)52.2356.8856.1654.2559.4357.34

Key Takeaways:

  • Performance scales monotonically with memory rank r at our primary 4-shot setting on both benchmarks (51.41 → 52.81 → 55.06 → 56.88 on TT, +5.47; 54.52 → 58.09 → 59.19 → 59.43 on Hindi, +4.91), confirming that higher-rank per-layer capacity (r = 64) is effectively utilized. These ablations further strengthen the robustness and usability of our framework.
AuthorsOpenReview ↗

Rebuttal: Part 3

Query 2:

(a) Clarification heterogeneous handling of HoloGram.

(b) Appendix C results

(c) Joint training relative to single-task adaptation.

(d) Data skewness and real world applicability

Response 2: We provide the response to this query by dividing it into four parts for more clarification:

(a) Here, in all the setups, training occurs only for the graph module and the layer-wise steered context. Therefore, it resolves heterogeneity as follows. If we consider for example the prior weight based alignment frameworks like PEFT with LoRA adapters, for each different setup, the whole model weight needs to be brought up to the GPU at inference time. This means that at the inference stage, if we have 9 different setups, then 9 complete model weights need to be actively present. But as with context steering based approaches (like with HoloGram), the complete model only needs to be brought to GPU only once, and then after that for each of the setup we only need to bring the steering weights that are powered by the GNN and layer specific memory weights only. That is how our approach handles heterogeneity in social media content moderation in a very low-resource setup. In short, we bring the base model once to the GPU and then only bring the context steering weights for each of the setup and use these only through PyTorch Hooks during inference time to cater different setups very easily. For better clarity, we will update these in the paper.

(b) In the Appendix C, our work naturally paves the path to this as a future work where we can have a monolithic framework for all types of content. This is the reason we also do not show the results of PMT/LIVE in Appendix and put these results in Appendix instead of the main content. Also, we have clearly mentioned in the corresponding section that current research constrained us to focus on binary classification (toxic vs non-toxic). Joint fine-grained classification leads to label mismatch (refer to Table 1; for eg: some dataset have fine-grained label as profane, while others don’t have and some have label dangerous while others don't) and that is why we were constrained to adjust only with binary classification. In future, we plan to revise the taxonomy and perform re-annotation so that fine-grained results can also be catered jointly. Further, to further bolster our argument, we run the results on PMT/LIVE. Compared to HoloGram for InternVL (which had the best results), for multilinguality (PMT, LIVE, HoloGram) triplet Macro-F1 gives (61.31, 66.05, 76.98) and for multimodality it gives (70.77, 53.77, 72.76). We can observe that HoloGram beats these methods even for joint training and evaluation.

(c) We have extended our ablation (please refer to the response to Query - 1) and have shown that joint training indeed works the best. Coupled with our already ablated results on Context injection in Table 7 and fully covered ablation of each component, we strongly believe that HoloGram will become an anchor of content moderation research in the coming years.

(d) We thank you a lot, and acknowledge the data skewness. These datasets are past works and mostly have content curated from existing social media platforms. However, we agree with you on data skewness and would endeavour to address this in the future datasets we build.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Rebuttal: Part 1

We thank the reviewer for very insightful and constructive reviews with multiple pieces of feedback that have significantly improved the robustness of our paper. Based on the extensive ablations and other experiments we have carried out, we are now even more confident in the real-world adaptation of HoloGram.

Query 1: Missing ablations

Response 1: We sincerely thank you that you took your time to go through the ablation which we have done in the Appendix. Based on your and other reviewers’ feedback, we ran extensive ablations to demonstrate that HoloGram as a whole is powered through GNNs as well as through the layer-specific memory steers. We present the results and corresponding observations/takeaways one by one below. Note that we also provide the significance test results when we provide response to your second query:

All results below report Macro-F1 Score across 2-shot, 4-shot, and 8-shot settings (k ∈ {2, 4, 8}) on Toxic Tags (TT) and HASOC (Hindi). The model is Qwen-VL with 2B parameters. Every ablation varies a single component at a time around our unified default configuration (Ours: knn = 6, λ = 0.8, L_CE + L_SCL, C = 10, r = 64, multi-layer memory injection). Bold = best at our setting; italics = ours when second-best.


1. Architecture, GNN & Baseline Ablations

VariantTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
(1) Permutation-invariant set encoder → MLP49.3155.5955.1955.7557.6254.31
(2) Top-k raw candidate vectors → mean → MLP53.2552.6053.3155.1253.5755.36
(3) Memory injected at last decoder layer only32.9236.8125.0638.9840.5431.59
(4) PMT (rank-64 M_l = A_l B_l per layer; no retrieval)*—53.20——44.46—
(5) LoRA adapter—52.12——43.43—
(6) Ours (Full GNN + multi-layer memory)52.2356.8856.1654.2559.4357.34

Note that PMT does not support retrieval or few-shot conditioning (no k); its single shot-independent score is placed in the central 4-shot column for direct comparison. Also, LoRA rank = 64; shot-independent baseline placed in the central 4-shot column.

Here are the key takeaways:

  • Graph relational modeling matters over pooling/sets: At our primary 4-shot setting, our GNN outperforms the permutation-invariant set encoder by +1.29 on TT (56.88 vs. 55.59) and +1.81 on Hindi (59.43 vs. 57.62), and outperforms mean-pooling + MLP by +4.28 on TT and +5.86 on Hindi (mean-pooling plateaus across shot counts rather than exploiting additional candidates).
  • Selective-layer memory injection does not perform better: Restricting memory injection to only the last decoder layer causes severe degradation across all shot counts (-20.07 on TT and -18.89 on Hindi at 4 shots).
  • Substantial gains over non-retrieval PEFT/memory baselines: Even when PMT uses per-layer rank-64 memory (M_l = A_l B_l), our retrieval-conditioned GNN achieves +3.68 over PMT and +4.76 over LoRA on TT, and +14.97 over PMT / +16.00 over LoRA on Hindi at 4 shots.

2. Training Objective (Loss) Ablation

Loss ConfigurationTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
loss_ce_only52.6654.9652.0956.3258.8254.64
loss_scl_only27.2127.2127.2135.4435.4435.44
loss_ce_and_scl (Ours)52.2356.8856.1654.2559.4357.34

Following are the inferred observations:

  • loss_scl_only lacks token/task supervision and collapses to a constant baseline (27.21 on TT, 35.44 on Hindi) regardless of shot count.
  • Combining L_CE + L_SCL consistently improves 4-shot performance (+1.92 on TT, +0.61 on Hindi) and prevents degradation when scaling to 8 shots (+4.07 on TT, +2.70 on Hindi over CE alone).
AuthorsOpenReview ↗

Rebuttal: Part 2

3. Graph & Retrieval Hyperparameter Sensitivity (knn, C, λ)

3a. Graph Edge Connectivity (knn)

knn EdgesTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
knn = 351.0155.4352.2454.3555.2353.04
knn = 450.2355.1656.4554.6659.1857.30
knn = 551.7355.5556.7452.2255.6551.05
knn = 6 (Ours)52.2356.8856.1654.2559.4357.34

3b. Number of Graph Nodes (C)

Nodes (C)TT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
C = 551.5953.3353.9253.7754.9953.87
C = 10 (Ours)52.2356.8856.1654.2559.4357.34
C = 2054.2957.4457.6154.8258.5352.52

3c. MMR Diversity Weight (λ)

MMR λTT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
λ = 0.2549.7454.0453.8153.5860.8357.61
λ = 0.550.3755.2454.8556.1658.8755.26
λ = 0.8 (Ours)52.2356.8856.1654.2559.4357.34

Key takeaways are as follows:

  • knn = 6 delivers the highest 4-shot score on both datasets (56.88 TT, 59.43 Hindi).
  • C = 10 is the optimal cross-dataset sweet spot: moving from C = 5 to C = 10 yields major gains at 4 shots (+3.55 TT, +4.44 Hindi), whereas C = 20 doubles graph size for only +0.56 on TT while hurting Hindi (-0.90 at 4 shots, -4.82 at 8 shots).
  • λ = 0.8 provides strong cross-dataset robustness without per-dataset tuning: it improves TT monotonically across 2, 4, and 8 shots (+2.84 over λ = 0.25 at 4 shots) while remaining competitive (second-best) on Hindi.

4. Memory Capacity (Rank r) Ablation

Memory Rank (r)TT (2 shots)TT (4 shots)TT (8 shots)Hindi (2 shots)Hindi (4 shots)Hindi (8 shots)
r = 846.9051.4152.7549.6654.5250.85
r = 1648.9052.8152.6551.4558.0955.32
r = 3249.9755.0656.1550.6359.1955.76
r = 64 (Ours)52.2356.8856.1654.2559.4357.34

Key Takeaways:

  • Performance scales monotonically with memory rank r at our primary 4-shot setting on both benchmarks (51.41 → 52.81 → 55.06 → 56.88 on TT, +5.47; 54.52 → 58.09 → 59.19 → 59.43 on Hindi, +4.91), confirming that higher-rank per-layer capacity (r = 64) is effectively utilized. These ablations further strengthen the robustness and usability of our framework.
AuthorsOpenReview ↗

Rebuttal: Part 3

Query 2:

(a) Clarification on the term ‘heterogenity’.

(b) Appendix C results and addition of PMT/LIVE baselines

(c) Joint training relative to single-task adaptation.

(d) Data skewness and real world applicability

Response 2:

(a) Here, in all the setups, training occurs only for the graph module and the layer-wise steered context. Therefore, it resolves heterogeneity as follows. If we consider for example the prior weight based alignment frameworks like PEFT with LoRA adapters, for each different setup, the whole model weight needs to be brought up to the GPU at inference time. This means that at the inference stage, if we have 9 different setups, then 9 complete model weights need to be actively present. But as with context steering based approaches (like with HoloGram), the complete model only needs to be brought to GPU only once, and then after that for each of the setup we only need to bring the steering weights that are powered by the GNN and layer specific memory weights only. That is how our approach handles heterogeneity in social media content moderation in a very low-resource setup. In short, we bring the base model once to the GPU and then only bring the context steering weights for each of the setup and use these only through PyTorch Hooks during inference time to cater different setups very easily. For better clarity, we will update these in the paper.

(b) This is the exact reason we put it in the Appendix and not in the main content. Our work naturally paves the path to this future work where we can have a monolithic framework for all types of content. This is the reason we also do not show the results of PMT/LIVE in Appendix as well. Also, we have clearly mentioned in the corresponding section that current research constrained us to focus on binary classification (toxic vs non-toxic). Joint fine-grained classification leads to label mismatch (refer to Table 1; for eg: some dataset have fine-grained label as profane, while others don’t have and some have label dangerous while others don't) and that is why we were constrained to adjust only with binary classification. In future, we plan to revise the taxonomy and perform re-annotation so that fine-grained results can also be catered jointly. Further, since you asked for the results of PMT/LIVE, we ran those experiments. Compared to HoloGram for InternVL (which had the best results), for multilinguality (PMT, LIVE, HoloGram) triplet Macro-F1 gives (61.31, 66.05, 76.98) and for multimodality it gives (70.77, 53.77, 72.76). We can observe that HoloGram beats these methods even for joint training and evaluation.

(c) We have extended our ablation (please refer to the response to Query - 1) and have shown that joint training indeed works the best. Coupled with our already ablated results on Context injection in Table 7 and fully covered ablation of each component, we strongly believe that HoloGram will become an anchor of content moderation research in the coming years.

(d) We thank you a lot, and acknowledge the data skewness. These datasets are past works and mostly have content curated from existing social media platforms. However, we agree with you on data skewness and would endeavour to address this in the future datasets we build.

Query 3:

(a) Value of C (20 or 10)?

(b) Implementation of cosine as edge weights in GATv2 and its processing.

(c) Statistical significance test

(d) Confusion over corrected parser and result discrepancy

Response 3:

(a) We apologise for the confusion. During resizing the paper to the page limit, one of the lines got deleted. Initially, C = 20, but after MMR, we selectively choose C=10 nodes. We will again add this line in the paper to reflect the final value of C after MMR-based diversity selection.

(b) Cosine values are directly fed as edge weights to the GATv2 layer. In the paper, we also point to the documentation of GATv2: https://pytorch-geometric.readthedocs.io/en/2.6.1/generated/torch_geometric.nn.conv.GATv2Conv.html. This documentation comprehensively describes the usage of cosine weights and how it is used by the GATv2 layer to learn better embedding representations. We would be happy to answer any future doubts you still might have about the implementation details of the GATv2 paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.