- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors proposed a "CodedNN" framework for improving robustness in neural network classification. Instead of the class labels being learned directly, an error-correcting code is learned instead. The model is trained to output different bits of the coded message, with the same network representation shared to predict each bit. The labels are recovered from the network output via one of two different processes: taking the first bits (which is a "plain-text" string describing the label) as the label or by using a belief propagation (BP) algorithm to determine the label using the full encoding. The authors state a result bounding the probability of a misclassification based on the covariances of the errors of each output channel. The authors then perform empirical validation with several datasets. Their results show mixed but generally positive performance compared to an unencoded baseline.
Critical strengths
- The potential gains from the method can be quite large compared to the baselines, and when the method performs worse than the baseline the degradation is minor.
- The improvements of the method with regard to robustness to feature noise as shown in Figure 2 are substantial and interesting.
Critical weaknesses
- This paper suffers from severe issues with clarity.
- The discussion in Section 3.3 needs a lot more explanation. As it stands, equation (1) is just the decomposition of the variance and mean of the sum of dependent variables, equation (2) is just a restatement of Cantelli's inequality, and equation (3) is an explicit formula for the covariance of the errors when the scores are Gaussian. No interpretation of these equations are provided. It would be helpful if specific values of the covariances and means were given such that equation (2) becomes a useful bound on the failure probability. Furthermore, clear insights that can be learned from the Gaussian case should be given. No further discussion of these results appear later in the paper, and their relevance to the rest of the paper is unclear. These results bound the error of a standard "nearest code-word" decoding mechanism, which does not appear to be used to recover the labels elsewhere in the paper.
- Some important notation is not properly defined in the paper. The difference between and in Section 5.2 is not explained. in the caption of Table 2 is not defined.
- The BP procedure for decoding is not described in the paper.
- The paper makes vague and unsubstantiated claims.
- Section 6.1 does not properly answer the question introduced in its title "Does Parity Structure Matter?". There is no variation of parity structure for a fixed dataset presented in Panel A of Table 2. The difference in performance may be a result of varying the datasets.
- In lines 65--67 the authors state as justification for why their method works that "Parity-structured targets can alter the representation learned during training even when only the systematic outputs are used at test time. Separately, an explicit decoder can exploit redundant predictions at inference." This explanation is repeated throughout the paper (for example, in the first sentence of the conclusion). I find these claims to be extremely vague, and they do not appear to correspond to any precise claim made in the paper. The first of these claims is especially suspect, as nowhere in the main text of the paper do the authors compare characteristics of the learned representations obtained with and without their method. Because the coding process is itself a linear function of the labels, it is possible that the training process of this method merely recovers the original network, but with the last layer modified to output the encoding instead of the labels.
Questions for authors
I would like the authors to clarify the following:
- What does the statement in lines 65--67 mean?
- What is the significance of the analysis in Section 3.3 to the rest of the paper?
3 comments
Thank you for your response. I appreciate the clarification on the notation and the methods.
- Can the authors state where the BP decoder procedure is described in the new draft of the paper? I was not able to find it when reviewing the updated document.
- I find the description of Table 3 unclear. Is it correct that is the result of using a simple repetition encoding and is the result of using a parity encoding? If so, this is concerning, as the column seems to indicate that the error-correcting code produces worse results than repetition in general.
- The utility of the bound in Section 3.3 is still not explicit in the paper. Is there a setting where the bound can be computed more easily than the true error rate?
Thank you for the follow-up. We address each point below and have revised the paper accordingly.
- BP procedure. Section 3.4 (Prediction and Optional Decoding) introduces the optional BP path. Section 4, under "Codes and training", specifies it: BP takes the raw code-aligned logits as channel evidence and runs 15 sum-product iterations with damping 0.8. We then read the posterior scores at the systematic label positions and apply thresholds fitted on clean validation data. Section 6.2 compares BP with direct readout, and Appendix D.4 reports the full 24-seed decoder results, including the corrected check-node update.
- Table 3. Yes. Arm C uses label repetition and arm D uses LDPC parity, with the same output width, padding, and systematic-label loss. All arms use raw systematic readout, so no decoder is applied: the table compares auxiliary training targets, not error-correction performance. The D−C column reports LDPC minus repetition in mean micro-F1. Repetition scores higher in four of the six dataset and noise settings, by up to 3.02 percentage points on clean Enron. The two positive differences for LDPC are small: +0.04 on clean Emotions and +0.27 on corrupted Enron. We agree that Table 3 shows no advantage for the tested LDPC targets over repetition. Repetition is itself a simple code, so this comparison tests whether LDPC parity structure adds value beyond repeating the labels. Relative to arm B, the matched head without auxiliary targets, both repetition and LDPC targets improve mean micro-F1 in four of six settings (C−B equals (D−B) minus (D−C)). Section 6.1 now states this.
- When the bound is useful. The bound needs only the mean and variance of the error count, which follow from marginal and pairwise error statistics, so it can be computed without running a decoder. Under the Gaussian-margin reference model, for example, these moments require only univariate and bivariate Gaussian probabilities, and the resulting bound holds conditional on that pairwise model. Because the error count is an integer between 0 and N, the same two moments also give sharp lower and upper bounds through a small linear program. The upper bound never exceeds Cantelli. On clean CIFAR-10, conditional on the measured moments, the interval for hard-decoder failure is 2.8–50.9%, with 27.2% observed failure; Cantelli gives 63.0%. Under the tested corruptions, the lower bounds certify failure rates of at least 41.9–57.4%, while the upper bounds remain near 100%. The moments therefore show that hard decoding is unreliable under these corruptions without evaluating the decoder. When decoded outcomes are available, the failure rate can be measured directly. Section 3.3 now states this, and a new table in Appendix F reports the per-condition results. The guarantee applies to hard bounded-distance decoding; we evaluate BP and soft readout empirically.
Thank you for identifying where the analysis and mechanism claims needed more support. We now vary the auxiliary targets within each dataset, explain the decoder procedure, and give a concrete test of when the correction-radius bound is informative.