- Rating
- 4
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper makes the improvements in activation shaping based methods. For the ASH/SCALE paper, the negative activations have detrimental effect on scaling. These methods dont generalize to architectures (ex. ViTs) which produce negative activations. Those methods primarily work well on architectures with non-negative activations such as ResNets and using batch normalization.
The paper proposes two fixes in the heuristics formula used by ASH / SCALE. Instead of using the original activations, they replace that with absolute activations. Hence, it measures relative strength of absolute top-k activations with respect to absolute total activations. Apart from that, paper proposes anti-coherence fix which determines disagreement of constituent dimensions with the prediction. ID samples have relatively less disagreement among dimensions while OOD samples have more anti-coherence and the scaling factor is penalized more. Combining these two heuristics, paper improves over the scaling the activation for separable energy score between ID and OOD.
Critical strengths
The findings of the paper is genuinely novel. The absolute fix is very simple, just two operation in existing scaling heuristics. And, the anti coherence term introduced by the paper is also simple and genuinely novel to the best of my knowledge. It's not been used in the literature so far to my understanding. The additive contribution of the paper doesn't introduce non-trivial computational overhead.
The performance of the method across architectures is robust. The paper makes meaningful advance in scaling based OOD detection. The paper has good presentation. There's no concern of hyperparameter sensitivity too.
Critical weaknesses
There are no critical weakness as such in the paper. This paper is a clear accept for me. However, a few clarifications could be useful.
Performance seems largely attributed to the absolute fix. How much does Anti-coherence alone contribute to the OOD detection? Does anti-coherence fix alone added to SCALE bring improvement over the average performance across all architectures in ImageNet + CIFAR datasets?
Or does anti-coherence contribute only when absolute fix is present? A discussion on this would be useful. And, could authors separate its result alone on relu-activations architectures vs others?
Furthermore, I am curious whether both of the components of the paper ( absolute fix and anti coherence) can be applied to AdaSCALE for performance boost similarly to SCALE? Results on these would be interesting.
The paper seems to have two hyperparameters which needs exhaustive tuning or could authors provide discussions on that? How much would the average performance become if lambda is fixed to one value (determined from one settting) across all settings? Does it still outperform all methods on average performance?
What do authors mean by SCALE + Relu. Is this LTS?
Questions for authors
Please see weaknesses.