- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HOLOGRAM is a graph-conditioned activation-steering method for toxic-content classification across languages, modalities, and label granularities. For each query, it builds a neighborhood of training examples, applies a GATv2 encoder, selects representative supports, and converts them into query-conditioned low-rank context vectors injected into every frozen VLM decoder layer. The paper evaluates Qwen3-VL and Intern3.5-VL at 2B and 8B scales on HateXplain, Amharic, HASOC, BHM, and ToxicTags in binary and fine-grained settings.
Critical strengths
- The method gives a coherent interface for heterogeneous moderation tasks. Query-conditioned graph support and per-layer low-rank injection are a technically combination: the VLM remains frozen, while the retrieved evidence can influence different decoder layers without being appended as a long prompt.
- The experiments cover two VLM families, two model sizes, multilingual text, multimodal memes, and binary versus fine-grained labels. The comparisons with both prompt-based methods and activation-steering baselines, together with the combined-suite and random-context analyses.
Critical weaknesses
- Appendix D.4 replaces support representations with Gaussian vectors only at inference in an already trained model. This demonstrates sensitivity to the learned input distribution, but does not establish the benefit of graph structure. Since the query itself enters the graph, a trained query-only encoder or a support-set encoder without edges could explain part of the gain. Neither control is reported.
- The baseline comparisons change support selection and adaptation together. ICL uses similarity-ranked examples, whereas HOLOGRAM adds clustering, diversity selection, contrastive supervision, and layer-wise adaptation. Their individual contributions remain unclear. Training for three rather than five epochs also does not establish lower cost: each HOLOGRAM training query requires separate VLM passes for three shot counts (Section 4.5).
Questions for authors
- How does HOLOGRAM compare with trained query-only and permutation-invariant support-set encoders using the same context builder and layer-wise steering, with comparable supervision and parameter budgets?
- Do the gains persist when ICL uses the same support-selection policy and PMT/LIVE receive comparable parameter and training-compute budgets? Please report trainable parameter counts, VLM training passes, and measured training and inference costs.
10 comments
Rebuttal: Part 2
3. Graph & Retrieval Hyperparameter Sensitivity (knn, C, λ)
3a. Graph Edge Connectivity (knn)
knn Edges | TT (2 shots) | TT (4 shots) | TT (8 shots) | Hindi (2 shots) | Hindi (4 shots) | Hindi (8 shots) |
|---|---|---|---|---|---|---|
knn = 3 | 51.01 | 55.43 | 52.24 | 54.35 | 55.23 | 53.04 |
knn = 4 | 50.23 | 55.16 | 56.45 | 54.66 | 59.18 | 57.30 |
knn = 5 | 51.73 | 55.55 | 56.74 | 52.22 | 55.65 | 51.05 |
knn = 6 (Ours) | 52.23 | 56.88 | 56.16 | 54.25 | 59.43 | 57.34 |
3b. Number of Graph Nodes (C)
Nodes (C) | TT (2 shots) | TT (4 shots) | TT (8 shots) | Hindi (2 shots) | Hindi (4 shots) | Hindi (8 shots) |
|---|---|---|---|---|---|---|
C = 5 | 51.59 | 53.33 | 53.92 | 53.77 | 54.99 | 53.87 |
C = 10 (Ours) | 52.23 | 56.88 | 56.16 | 54.25 | 59.43 | 57.34 |
C = 20 | 54.29 | 57.44 | 57.61 | 54.82 | 58.53 | 52.52 |
3c. MMR Diversity Weight (λ)
MMR λ | TT (2 shots) | TT (4 shots) | TT (8 shots) | Hindi (2 shots) | Hindi (4 shots) | Hindi (8 shots) |
|---|---|---|---|---|---|---|
λ = 0.25 | 49.74 | 54.04 | 53.81 | 53.58 | 60.83 | 57.61 |
λ = 0.5 | 50.37 | 55.24 | 54.85 | 56.16 | 58.87 | 55.26 |
λ = 0.8 (Ours) | 52.23 | 56.88 | 56.16 | 54.25 | 59.43 | 57.34 |
Key takeaways are as follows:
knn = 6delivers the highest 4-shot score on both datasets (56.88 TT, 59.43 Hindi).C = 10is the optimal cross-dataset sweet spot: moving fromC = 5toC = 10yields major gains at 4 shots (+3.55 TT, +4.44 Hindi), whereasC = 20doubles graph size for only+0.56on TT while hurting Hindi (-0.90 at 4 shots, -4.82 at 8 shots).λ = 0.8provides strong cross-dataset robustness without per-dataset tuning: it improves TT monotonically across 2, 4, and 8 shots (+2.84 overλ = 0.25at 4 shots) while remaining competitive (second-best) on Hindi.
4. Memory Capacity (Rank r) Ablation
Memory Rank (r) | TT (2 shots) | TT (4 shots) | TT (8 shots) | Hindi (2 shots) | Hindi (4 shots) | Hindi (8 shots) |
|---|---|---|---|---|---|---|
r = 8 | 46.90 | 51.41 | 52.75 | 49.66 | 54.52 | 50.85 |
r = 16 | 48.90 | 52.81 | 52.65 | 51.45 | 58.09 | 55.32 |
r = 32 | 49.97 | 55.06 | 56.15 | 50.63 | 59.19 | 55.76 |
r = 64 (Ours) | 52.23 | 56.88 | 56.16 | 54.25 | 59.43 | 57.34 |
Key Takeaways:
- Performance scales monotonically with memory rank
rat our primary 4-shot setting on both benchmarks (51.41 → 52.81 → 55.06 → 56.88on TT, +5.47;54.52 → 58.09 → 59.19 → 59.43on Hindi, +4.91), confirming that higher-rank per-layer capacity (r = 64) is effectively utilized. These ablations further strengthen the robustness and usability of our framework.
Rebuttal: Part 3
Query 2: Parameters and inference efficiency
Response 2: First of all, we thank you for pointing this out. Below we present the trainable parameter efficiency of our framework:
| Model | GNN (%) | Learnable Memory (%) | Total (%) |
|---|---|---|---|
| Qwen3-VL 2B | 0.444% | 0.487% | 0.931% |
| InternVL3.5 2B | 0.402% | 0.441% | 0.843% |
| Qwen3-VL 8B | 0.108% | 0.277% | 0.384% |
| InternVL3.5 8B | 0.111% | 0.284% | 0.395% |
As we can observe, HoloGram adds less than 1% total overhead (0.843%–0.931% at 2B and 0.384%–0.395% at 8B). Also, moving from 2B to 8B more than halves the relative total overhead (0.931% → 0.384% for Qwen3-VL; 0.843% → 0.395% for InternVL3.5), with the GNN share dropping nearly 4× (~0.40–0.44% → ~0.11%). This further shows the efficiency of trainable parameters. Most importantly, for all setups, the model is loaded only once and PyTorch’s Hook is applied for inference-time steering using GNN and learnable memory parameters for different setups, which tunnel down to a highly efficient system.
Finally, we also present inference stats for Qwen - VL 2B model:
| Dataset | Method | k | Prompt Tokens | Generated Tokens | Generation (ms) | Peak GPU Memory (GiB) |
|---|---|---|---|---|---|---|
| Hindi (T, 4) | Zero-shot | 0 | 364 | 2.9 | 84.3 | 8.15 |
| ICL (cosine top-k) | 8 | 1,584 | 3.0 | 142.9 | 8.47 | |
| HoloGram | 8 | 364 | 2.6 | 100.2 | 8.81 | |
| ToxicTags (M, 4) | Zero-shot | 0 | 671 | 2.5 | 115.9 | 8.37 |
| ICL (cosine top-k) | 8 | 3,899 | 2.9 | 467.0 | 9.33 | |
| HoloGram | 8 | 671 | 2.7 | 150.7 | 10.05 |
Because HoloGram conditions via latent memory rather than stuffing retrieved examples into the context window, it keeps prompt length identical to zero-shot; reducing prompt tokens over 8-shot ICL by 4.35× on Hindi (364 vs. 1,584, -77.0%) and 5.81× on multimodal ToxicTags (671 vs. 3,899, -82.8%). Further, it is up to 3.10× faster inference than ICL:. On multimodal ToxicTags (k = 8), HoloGram cuts generation latency from 467.0 ms to 150.7 ms (3.10× faster, -67.7%), and on text Hindi from 142.9 ms to 100.2 ms (1.43× faster, -29.9%), adding only 15.9–34.8 ms over zero-shot. This leads only to a modest memory footprint. Peak GPU memory increases by only +0.34 GiB on Hindi (8.81 vs. 8.47 GiB) and +0.72 GiB on ToxicTags (10.05 vs. 9.33 GiB) relative to 8-shot ICL.
Rebuttal: Part 1
We thank the reviewer for very insightful and constructive reviews with multiple feedbacks that have significantly bolstered the robustness of our paper. Based on the extensive ablations and other experiments we have carried out, we are now even more confident in the real world adaptation of HoloGram.
Query 1: Missing ablations
Response 1: We sincerely thank you that you took your time to go through the ablation which we have done in the Appendix. Based on your and other reviewers’ feedback, we ran extensive ablations to demonstrate that HoloGram as a whole is powered through GNNs as well as through the layer-specific memory steers. We present the results and corresponding observations/takeaways one by one below:
All results below report Macro-F1 Score across 2-shot, 4-shot, and 8-shot settings (
k ∈ {2, 4, 8}) on Toxic Tags (TT) and HASOC (Hindi). The model is Qwen-VL with 2B parameters. Every ablation varies a single component at a time around our unified default configuration (Ours:knn = 6,λ = 0.8,L_CE + L_SCL,C = 10,r = 64, multi-layer memory injection). Bold = best at our setting; italics = ours when second-best.1. Architecture, GNN & Baseline Ablations
M_l = A_l B_lper layer; no retrieval)*Note that PMT does not support retrieval or few-shot conditioning (no
k); its single shot-independent score is placed in the central 4-shot column for direct comparison. Also, LoRA rank = 64; shot-independent baseline placed in the central 4-shot column.Here are the key takeaways:
56.88vs.55.59) and +1.81 on Hindi (59.43vs.57.62), and outperforms mean-pooling + MLP by +4.28 on TT and +5.86 on Hindi (mean-pooling plateaus across shot counts rather than exploiting additional candidates).M_l = A_l B_l), our retrieval-conditioned GNN achieves +3.68 over PMT and +4.76 over LoRA on TT, and +14.97 over PMT / +16.00 over LoRA on Hindi at 4 shots.2. Training Objective (Loss) Ablation
loss_ce_onlyloss_scl_onlyloss_ce_and_scl(Ours)Following are the inferred observations:
loss_scl_onlylacks token/task supervision and collapses to a constant baseline (27.21 on TT, 35.44 on Hindi) regardless of shot count.L_CE + L_SCLconsistently improves 4-shot performance (+1.92 on TT, +0.61 on Hindi) and prevents degradation when scaling to 8 shots (+4.07 on TT, +2.70 on Hindi over CE alone).