- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how different output coding strategies affect learned representations under a shared low-dimensional bottleneck, providing theoretical analyses that connect code-dependent feature selection with classification error and Gaussian width. Experiments on Fashion-MNIST and CIFAR-100 show that output coding can significantly affect classification performance.
Critical strengths
-
The paper provides several theoretical results explaining how output coding affects representation learning under a shared bottleneck. I think the connection between classification error and Gaussian width is interesting and provides a useful perspective on understanding representation learning.
-
The paper addresses a well-defined problem and provides mathematical analysis to support its main findings. I also think the authors do a good job of explaining why conventional code properties may not be sufficient to characterize the learned representations.
Critical weaknesses
-
My main concern is that the experimental evaluation is relatively limited. Although the theoretical analysis might be interesting, the experiments are mainly conducted on Fashion-MNIST and CIFAR-100 using frozen features. I think more experiments on modern architectures with end-to-end representation learning would make the paper much more convincing.
-
Another concern is that I find this paper quite difficult to read. I think the writing is rather dry and not very engaging. In the introduction, there are too many technical terms without sufficient explanation or illustration. I am also not sure whether just the first two paragraphs in the introduction are enough to clearly position this paper within the existing research. At least for me, I find it difficult to understand the background and motivation of this work, which makes it hard to properly assess its value and significance. In fact, I still do not clearly see the value of this work.
-
In line 126, a period is missing after the word "matrix".
Questions for authors
-
Could you provide more experiments on modern architectures with end-to-end representation learning? I think this would make the paper more convincing and help show whether the findings are useful in practice.
-
Could you make the paper easier to read, especially the introduction? I think more background and clearer explanations of the main ideas would help readers better understand the motivation and contributions of this work.
-
Could you explain more clearly the main value of this work compared with existing methods? I am still not fully convinced why these theoretical findings are important and how they could be useful in practice.
2 comments
I appreciate the authors’ efforts to add end-to-end learning experiments and broader evaluations, and to improve the clarity of the paper’s intuition, motivation, contributions, and practical significance. These changes address some of my original concerns. However, the new experiments and substantial rewriting take the paper well beyond its original scope. I feel that it would need another full round of review, so I will maintain my original score.
Thank you for your constructive feedback on expanding the empirical evaluation, improving the Introduction, and clarifying the value of the work. Below, we address these points directly and identify the corresponding changes in the manuscript.
1. End-to-end learning and broader evaluation
Following your suggestion, we added an end-to-end CIFAR-100 experiment with a CIFAR-style ResNet-18 trained from scratch through a shared rank-16 bottleneck (Section 6 and Supplementary Section S12). Using 10,000 training images, 30 epochs, and three paired seeds, coding achieves 81.84% mean test error versus 83.39% for one-hot regression, both with validation-selected projected LDA decoding. Softmax with native decoding achieves 59.95%. This extends the comparison to jointly learned features and places the coding results alongside a strong cross-entropy baseline. The direct test of our subspace predictions remains the separate linear experiment described below.
Beyond ResNet-18, we evaluate frozen features from DINOv2-small, a self-supervised vision transformer, on Fashion-MNIST. At rank four, coding reduces error by 13.77 percentage points relative to one-hot regression using identical training examples and a common Gaussian decoder (Appendix L, Table 9). Supplementary Section S10.6 also reports a small experiment with a trainable nonlinear encoder on Optdigits. To test the predicted directions directly, we trained factorized linear predictors from random initialization (Section 6, Figure 3, and Appendix M). At rank one and the largest sample size, N = 10 million, their mean overlaps with their respective code's predicted population direction are 99.92% and 99.93%, versus 0.07% with the alternative code's prediction (Table 11). These results distinguish close recovery of the population directions from the broader image comparisons.
2. Readability, intuition, and motivation
Regarding the Introduction, we now explain output coding before using a garment-classification example to motivate the bottleneck. Distinguishing shirts, trousers, and coats through a small set of shared features illustrates why class distinctions compete for capacity. Figure 1 then shows how two codes favor different directions. Section 2 presents a four-class example—cat, dog, car, and truck—and Figure 2 follows the path from coded labels through the shared representation to the final prediction. These examples provide an intuitive route into the mathematical argument.
We also define the main terms where they become relevant. The Introduction explains class distinctions as information that changes relative class scores; Section 2 introduces the coding metric through its role in weighting the training loss; and Section 3 explains Gaussian width through its connection to classification error. We have also corrected the missing period you identified after ‘matrix’.
3. Contribution and practical significance
Your question about the value of the work prompted us to sharpen its positioning in Related Work (Section 1.1). Earlier work reports other benefits of coding: Verma and Swami (NeurIPS 2019) find improved adversarial robustness and probability estimates, while Yang et al. (AAAI 2015) report more discriminative features from coded targets. Our analysis complements these empirical findings. Classical analyses relate codeword separation, binary prediction errors, and decoding to multiclass performance; Evron et al. (2023) study assignment effects through the difficulty of the induced binary tasks. We isolate an additional effect of sharing a limited representation. Theorem 4 constructs codes with identical singular spectra, row and column Hamming-distance histograms, and Bayes errors for corresponding binary tasks, yet their retained subspaces have no nonzero direction in common and their optimal multiclass risks differ. Thus, the difference can persist even after controlling these code properties and binary-task difficulties.
The practical implication is that conventional code summaries alone cannot determine which class information a bottleneck retains. In our frozen-feature experiments, 32 row permutations share the same spectra and distance histograms, yet differences between codes averaged over training draws account for 76.5–97.0% of the observed candidate test-error variance on Fashion-MNIST and 39.3–52.5% on CIFAR-100 in the descriptive decomposition (Section 6; Appendix K.3, Table 5). We therefore recommend evaluating assignments on the target features at the intended bottleneck rank, with training examples and decoding controlled. One-hot regression and code whitening help identify the effect of nonuniform loss weighting (Proposition 5, Section 5). Sections 5–7 explain these practical implications, providing a basis for evaluating and developing codes that account for both data and representation capacity.