- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper picks 10 hidden-state units ("super neurons", SNs) inside a VLM using 3,000 labeled examples, and reads answers from their signed average before decoding. On seven yes/no VQA datasets, SNs beat the zero-shot model by 2.8 average F1 points on LLaVA-v1.5-7b and 16.0 on Qwen3-VL-8b, and guide sparse finetuning.
Critical strengths
-
A very simple, gradient-free readout of 10 units gives a real gain on balanced POPE. LLaVA's SNs raise both precision and recall there (Table 12), and gains hold at every Qwen3-VL size from 2b to 32b (Table 14).
-
The method is cheap and easy to inspect. Selection usually takes under 30 minutes on one GPU, and the POPE gains hold across five probing sets (Fig. 7).
Critical weaknesses
-
The headline comparison seems uneven, and this is the main reason for my rating. SNs are fit on 3,000 balanced labeled examples, but the base model is zero-shot and the other baselines see far fewer examples (App. A.3). With no dense linear probe or re-tuned threshold on the base model's yes/no score, the gain may mostly reflect the 3,000 labels, not "underutilized" signal.
-
Much of the gain seems to come from avoiding the base models' answer bias. For Qwen, about 90% of the average F1 gain comes from both InstaOrder splits and CLEVR, where Qwen mostly gives the same answer (Table 13). For LLaVA, average accuracy rises 14.0 points but F1 only 2.8, which matches a fix of its habit of answering yes too often (Table 15a).
-
The step beyond earlier work seems small. SNs act like a sparse probe with plus-or-minus-one weights, a modest step from sparse attention vectors (SAVs, Mitra et al. [1]). Skill neurons (Wang et al. [2]) and sparse probing (Gurnee et al. [3]) are also close, but the paper cites them without comparing.
-
The claim that SNs challenge the view that VLMs lack 3D understanding seems overstated for real images. On InstaOrder (Occ.), Qwen's SNs get 33.7 F1, not far above the 25.7 of a 3-shot run that answers yes nearly every time (Table 11). Their 68.4% accuracy is also below the base model's 84.6%, which comes from mostly answering no.
-
The 27.8× speedup seems to come mostly from the decoding setup. Qwen generates up to 128 tokens with no stop strings, averaging 80.72 tokens per POPE question (Table 4). Reading its first-token yes/no logits, as App. E already does for confidence, would need only the same prefill as SNs.
-
The generalization and finetuning claims rest on narrow tests, such as one POPE-to-VOC transfer (Table 2) and one finetuning task on LLaVA. The LoRA and SFT baselines keep default settings (App. A.4), so their collapse on MM-Vet may not be a fair contrast. In Table 5, 31.1% accuracy and 42.0 F1 on InstaOrder (Occ.) do not seem possible together at its yes rate of about 15%.
References:
[1] Chancharik Mitra et al. Enhancing few-shot vision-language classification with large multimodal model features. ICCV, 2025.
[2] Xiaozhi Wang et al. Finding skill neurons in pre-trained transformer-based language models. EMNLP, 2022.
[3] Wes Gurnee et al. Finding neurons in a haystack: Case studies with sparse probing. TMLR, 2023.
Questions for authors
Questions:
- On the same 3,000 samples, how do SNs compare with a dense logistic-regression probe, SAVs [1], and the base model's yes/no margin under a re-tuned threshold? If SNs beat the threshold baseline by a good margin and come close to the dense probe, I would raise my rating.
- Could you report each validation set's yes rate, a balanced metric such as AUROC, and standard multiple-choice accuracy on A-OKVQA and ScienceQA? Gains that hold under these would support the spatial and multiple-choice claims.
- Does SN selection work as well after each layer's hidden states are rotated at random? A large drop would show that single units matter and would raise my view of the novelty.
- Can SN inference stop at the deepest selected layer, and how much time does that save over reading the base model's first-token logits? A real saving would make the speedup an SN contribution.
Suggestions:
- The shuffle test (Fig. 3) shows SNs use both image and text, but a test on subtler biases like object co-occurrence would say more.
- Reporting spread over several probing sets for all seven datasets, not only POPE, would make the smaller gains easier to judge.
4 comments
Rebuttal reply 2/2
4. Efficient inference using super neurons
Reviewer JzEr is right in thinking that super neurons can be stopped right at the deepest selected layer.
To measure this, we constrain the search of super neurons up until a given layer and probe the units up until that layer. We find that we can significantly reduce the runtime of the model by exiting inference at that given layer, while still staying within 0.5 AUROC of full depth super neurons. This provides substantial gains of 1.38× on Qwen and 1.79× on LLaVA, averaged over 7 datasets. We show a case for Pope in C.14 (fig. 17).
5. Spread of the gains
Following your suggestion, we compute the full sample ablation over a number of probing samples from 10 to 3000 on 5 randomly sampled probing sets and report the 95% confidence interval on these runs. This helps assessing the variation in results.
We observe that super neurons outperform the model most of the time when probed at 100 samples. At 3000 samples, the confidence interval tightens around our reported measures. You can find the details results for both models on each datasets in appendix C.12 (figs 15 and 16).
6. MCQ results
Following your suggestion, we report the results of the standard MCQ evaluation, where we feed the complete prompt to the VLM.
We also design a simple approach to make super neurons work in this setting. Assuming (ex. If the MCQ contains A, B, C, and D, then ) possible classes in the dataset, we compute sets of super neurons, each responsible for answering that particular letter.
Results in table 13 and analyses in C.3 highlight that super neurons are still on par or better than the base model (L. 1236). This substantiates our claims that super neurons are effective on MCQ data.
We apologize for not covering the co-occurrence experiment, but due to the high volume of experiments requested for this rebuttal, we preferred prioritizing more critical experiments (such as the comparison to other baselines), which were also reported by other reviewers. We will try our best to update the rebuttal before the deadline to include this experiment, otherwise, we will include it in the final version of the manuscript.
We would like to thank reviewer JzEr once again for their thoughtful and useful feedback. We believe that their comments were instrumental to improve the quality of the manuscript.
Thanks for running the extra experiments. On the same 3K samples, SNs now outperform the re-thresholded yes/no logits on both models and come close to the dense probe. That addresses my main concern, so I’m raising my score from 2 to 3.
I’d still soften the claims about 3D understanding and MCQ performance. Re-thresholding alone seems to explain about 60–70% of the gain over the zero-shot model. The clearest improvements are on InstaOrder and a few LLaVA datasets, while the standard MCQ gains are only 0.0–0.5 points.
The 27.8× speedup in the abstract is measured against full autoregressive decoding (Table 4), while Fig. 17 shows 1.5–2.7× over first-token logits, so I’d mention the latter as well. I’d also double-check Table 18. In 8 of the 14 rows, the rotated SNs predict "no" for almost every sample, and the A-OKVQA numbers match exactly across models.
Finally, lines 129–131 should be corrected: skill neurons also use each neuron’s mean activation as a threshold and keep the better-performing direction.
Final details fixes. Thank you
Dear reviewer JzEr,
Thank you for your prompt reply, we appreciate that we could mitigate the concerns you had.
Softening the claims about 3D understanding and standard MCQ. We understand your concern, we have updated the PDF to reflect these softened claims.
27.8x speedup gain. We clarify here that these gains come from the comparison between the normal autoregressive inference of Qwen3-VL-8b and its super neurons on Pope. You are right in pointing out that Fig. 17 computes the runtime of super neurons against the calibrated yes/no logit classification baseline. This is to provide a more challenging baseline.
We have updated the abstract to reflect both these settings.
Rotation experiments. Thank you for pointing this out to us, we re-ran the rotation experiment under 10 random seeds and double checked the A-OKVQA results. We have updated table 18 with the correct results and the experimental description to reflect the seeding setup.
Skill neuron addition in the related works section. Thank you for this additional detail. We have added this suggestion in the manuscript.
We would like to thank you once again for the quality of your review and the promptness of your replies. We believe your suggestions made the manuscript more comprehensive and substantially stronger.
Authors of "Super Neurons Answer Before VLMs Speak".
Rebuttal reply 1/2
Dear reviewer JzEr,
thank you for your informative and thoughtful comments on our super neurons manuscript. We are pleased that you found our approach simple and efficient.
We updated the manuscript with the detailed answers to your questions. We colored the amended text in orange for you. We nonetheless concisely reply to your questions below.
1. Comparison with more baselines
Following your suggestion, we compare our super neurons to more baselines, notably:
1.1. Logit classification from the calibrated LLM last layer's hidden state
The macro-average confirms that super neurons, even when k=1 outperform the baseline by 3.0 AUROC score on average for LLaVA-v1.5-7b and 1.4 AUROC score for Qwen3-VL-8b.
We provide all the details in sec. C.6 (Fig. 10 for the AUROC macro-average) of the appendices and detail how we construct that baseline in appendix A.3, L. 1031.
1.2. Sparse probing
On the mean-diff. variant, which is most competitive, super neurons are on average 0.71 AUROC score ahead for LLaVA-v1.5-7b and 0.84 AUROC score ahead for Qwen3-VL-8b at k=100.
We invite you to refer to section C.6 of the appendices, where we analyze these baselines.
1.3. Traditional linear probing
This baseline mostly serves as an upper bound since it leverages many learnable parameters, while ours is completely gradient-free. None of the assessed baselines are able to outperform it, yet, ours comes the closest. Please refer to appendix C.6 for more information.
1.4. Budget-aligned SAV baseline
We re-evaluate SAVs under the same sample budget as our super neurons (using N=3000).
In supplementary C.5 (most specifically, on Fig. 10), we show that super neurons outperform AUROC scores of SAVs most of the time, despite only leveraging a handful of scalars. Moreover, SAV relies on orders of magnitude larger representations since it uses tensor clustering.
1.5. Main manuscript
To help reader understand how we position against all baselines, we report the macro-average of super neurons against SAVs, sparse probing, linear probing and the classification from the logits of the model in Fig. 8 of the main manuscript and comment it on L. 423.
2. More statistics
We report the ROC curve and the AUROC score of the base model's logits and our super neurons on all datasets.
We visually observe that AUROC of super neurons is at or above the first-token logit margin of the VLMs. We comment these figures in details in supplementary materials C.11 (Fig. 14). Moreover, we measured the standard multiple-choice accuracy on A-OKVQA and ScienceQA. We find that super neurons match or slightly exceed the model’s own answer in accuracy (+0.0 to +0.5).
3. Super neurons extraction under hidden states rotations
Following your request, we select the super neurons after hidden states are rotated at random. At evaluation, we observe a sharp performance degradation in the performance of the probed units on all the datasets, for both models. We agree with the reviewer that this large drop shows that single units matter. You can find the results of this experiment in appendix C.13 (table 18).