A Blockwise Probability Loss Function for Multiclass Classification Learning

Binary loss functions are very popular in multiclass classification learning algorithms due to their low computational complexity. On the other hand, some of them, such as softmax cross-entropy (CE) and binary cross-entropy (BCE) may impose restrictions on class separability or algorithm convergence. In standard multiclass classification, cross‑entropy is typically used with one‑hot targets, while the BCE reduces to the case where all outputs of the learned function are treated separately. We observe that if binary losses consider the whole output as a collection of separate binary outputs, they do not fully explore the extrinsic information from other components during training. Motivated by that, we present a new blockwise loss function that satisfies three basic properties: it is (i) not separable into binary subproblems, (ii) well defined for associating classes with binary vectors, which we refer to as binary code assignments, and (iii) obtained by comparing the output to the true class representation as well as to all other classes. We show that the proposed loss reduces to CE when one-hot coding is applied, while it reduces to a scaled version of BCE when property (iii) is simplified such that the loss compares the output only with the true class representation. Furthermore, when one-hot coding is replaced with error correcting output codes (ECOCs), the proposed loss achieves accuracy that is equal to or higher than that of classical bitwise losses. Experiments were conducted on the CIFAR-100 and Tiny ImageNet datasets for various losses from the literature and demonstrate the advantages of the new blockwise probability (BP) loss in combination with ECOCs.