acceptodds
Under review as a conference paper at ICLR 2027

CodedNN: Coding-Theoretic Redundancy for Robust Neural Classification

Abstract

Neural classifiers can degrade substantially under input corruptions and perturbations to learned features. We investigate a simple way to improve reliability: encode labels into redundant binary outputs and train a shared-backbone network to predict the resulting codeword. Coded Neural Networks (CodedNN) preserve the original label coordinates in multilabel classification (MLC), while redundant parity coordinates provide additional supervision and can optionally be exploited by belief-propagation (BP) decoding; in single-label classification (SLC), classes are represented by redundant binary codewords and prediction is performed by soft codeword matching. A central difficulty is that learned output errors are strongly dependent, so the independent-channel abstraction underlying classical error-correcting codes does not accurately describe neural predictions. We therefore develop a dependence-aware analysis of aggregate codeword errors, including a distribution-free correction-radius bound and a Gaussian-margin reference model. Empirically, redundant outputs provide gains in both clean and corrupted regimes. Across 13 clean MLC benchmarks, raw CodedNN improves micro-F1 over a matched binary-cross-entropy predictor by 1.5 percentage points on average, while explicit decoding provides additional robustness in some corrupted regimes. On CIFAR-10-C, redundant 8-bit output representations improve mean corruption accuracy by up to 1.67 percentage points over softmax with essentially unchanged clean accuracy, and on ImageNet the coded model exceeds softmax by 12.2 points under the strongest tested feature perturbation. These results show that structured output redundancy can improve neural reliability even when classical coding assumptions fail, providing a framework for exploiting redundancy through both representation learning and decoding.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Robust neural classification, error-correcting codes, structured output redundancy, covariance conditioning, correlated prediction errors, multilabel classification, reliable AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oD8xOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors proposed a "CodedNN" framework for improving robustness in neural network classification. Instead of the class labels being learned directly, an error-correcting code is learned instead. The model is trained to output NN different bits of the coded message, with the same network representation shared to predict each bit. The labels are recovered from the network output via one of two different processes: taking the first CC bits (which is a "plain-text" string describing the label) as the label or by using a belief propagation (BP) algorithm to determine the label using the full encoding. The authors state a result bounding the probability of a misclassification based on the covariances of the errors of each output channel. The authors then perform empirical validation with several datasets. Their results show mixed but generally positive performance compared to an unencoded baseline.

Critical strengths
  1. The potential gains from the method can be quite large compared to the baselines, and when the method performs worse than the baseline the degradation is minor.
  2. The improvements of the method with regard to robustness to feature noise as shown in Figure 2 are substantial and interesting.
Critical weaknesses
  1. This paper suffers from severe issues with clarity.
    1. The discussion in Section 3.3 needs a lot more explanation. As it stands, equation (1) is just the decomposition of the variance and mean of the sum of dependent variables, equation (2) is just a restatement of Cantelli's inequality, and equation (3) is an explicit formula for the covariance of the errors when the scores are Gaussian. No interpretation of these equations are provided. It would be helpful if specific values of the covariances and means were given such that equation (2) becomes a useful bound on the failure probability. Furthermore, clear insights that can be learned from the Gaussian case should be given. No further discussion of these results appear later in the paper, and their relevance to the rest of the paper is unclear. These results bound the error of a standard "nearest code-word" decoding mechanism, which does not appear to be used to recover the labels elsewhere in the paper.
    2. Some important notation is not properly defined in the paper. The difference between Fcoded,rawF_{coded,raw} and Fover,rawF_{over,raw} in Section 5.2 is not explained. GBPG_{BP} in the caption of Table 2 is not defined.
    3. The BP procedure for decoding is not described in the paper.
  2. The paper makes vague and unsubstantiated claims.
    1. Section 6.1 does not properly answer the question introduced in its title "Does Parity Structure Matter?". There is no variation of parity structure for a fixed dataset presented in Panel A of Table 2. The difference in performance may be a result of varying the datasets.
    2. In lines 65--67 the authors state as justification for why their method works that "Parity-structured targets can alter the representation learned during training even when only the systematic outputs are used at test time. Separately, an explicit decoder can exploit redundant predictions at inference." This explanation is repeated throughout the paper (for example, in the first sentence of the conclusion). I find these claims to be extremely vague, and they do not appear to correspond to any precise claim made in the paper. The first of these claims is especially suspect, as nowhere in the main text of the paper do the authors compare characteristics of the learned representations obtained with and without their method. Because the coding process is itself a linear function of the labels, it is possible that the training process of this method merely recovers the original network, but with the last layer modified to output the encoding instead of the labels.
Questions for authors

I would like the authors to clarify the following:

  1. What does the statement in lines 65--67 mean?
  2. What is the significance of the analysis in Section 3.3 to the rest of the paper?
3 comments

Thank you for identifying where the analysis and mechanism claims needed more support. We now vary the auxiliary targets within each dataset, explain the decoder procedure, and give a concrete test of when the correction-radius bound is informative.

  • Auxiliary targets and representations (§§1 and 6.1, Table 3). Auxiliary targets contribute gradients through the shared backbone during training. XOR parity is linear over GF(2), while a fixed linear map of real-valued label scores cannot generally recover it. A nonlinear head could still learn the parity without improving the backbone. We therefore focus on the measured predictions. The matched repetition control shares width and systematic losses with LDPC, and LDPC shows no consistent absolute micro-F1 advantage over it.
  • Dependence and the bound (§3.3, Appendix F). The variance decomposition makes the covariance contribution to the total error count explicit. In the Gaussian reference model, positive pairwise margin correlation produces positive error covariance at fixed marginal error rates. This pair contributes more to the error-count variance than under independence. On clean CIFAR-10, a hard decoder with correction radius one has 27.16% observed error and a 63.02% mean plug-in bound. Under the tested corruptions, the mean error count exceeds two, so the bound is trivial. We use this hard decoder for the diagnostic. The primary results use raw readout, BP, or soft codeword scoring.
  • Notation and BP (§§4 and 6, Table 2). The original §5.2 material now appears in §6.1. Δ_code(σ) is the raw CodedNN minus overcomplete micro-F1 gap at noise level σ. R_code(σ) subtracts the clean gap. G_BP is the BP minus raw micro-F1 gain, and ΔBER is post-BP minus raw codeword BER. BP starts from the raw code-aligned logits, runs 15 sum-product iterations with damping 0.8, and reads the posterior scores on the systematic coordinates using thresholds fit on clean validation data.
  • Delicious (§3.2, Appendix D). The 0.451 score-residual correlation was computed on the test split, which the revised caption now states. The original Gaussianity script read the wrong checkpoint key. The replacement uses the trained weights and validation residuals. Appendix D also reports the hard-bit error-count diagnostics.

Thank you for your response. I appreciate the clarification on the notation and the methods.

  • Can the authors state where the BP decoder procedure is described in the new draft of the paper? I was not able to find it when reviewing the updated document.
  • I find the description of Table 3 unclear. Is it correct that CC is the result of using a simple repetition encoding and DD is the result of using a parity encoding? If so, this is concerning, as the column D−CD - C seems to indicate that the error-correcting code produces worse results than repetition in general.
  • The utility of the bound in Section 3.3 is still not explicit in the paper. Is there a setting where the bound can be computed more easily than the true error rate?

Thank you for the follow-up. We address each point below and have revised the paper accordingly.

  • BP procedure. Section 3.4 (Prediction and Optional Decoding) introduces the optional BP path. Section 4, under "Codes and training", specifies it: BP takes the raw code-aligned logits as channel evidence and runs 15 sum-product iterations with damping 0.8. We then read the posterior scores at the systematic label positions and apply thresholds fitted on clean validation data. Section 6.2 compares BP with direct readout, and Appendix D.4 reports the full 24-seed decoder results, including the corrected check-node update.
  • Table 3. Yes. Arm C uses label repetition and arm D uses LDPC parity, with the same output width, padding, and systematic-label loss. All arms use raw systematic readout, so no decoder is applied: the table compares auxiliary training targets, not error-correction performance. The D−C column reports LDPC minus repetition in mean micro-F1. Repetition scores higher in four of the six dataset and noise settings, by up to 3.02 percentage points on clean Enron. The two positive differences for LDPC are small: +0.04 on clean Emotions and +0.27 on corrupted Enron. We agree that Table 3 shows no advantage for the tested LDPC targets over repetition. Repetition is itself a simple code, so this comparison tests whether LDPC parity structure adds value beyond repeating the labels. Relative to arm B, the matched head without auxiliary targets, both repetition and LDPC targets improve mean micro-F1 in four of six settings (C−B equals (D−B) minus (D−C)). Section 6.1 now states this.
  • When the bound is useful. The bound needs only the mean and variance of the error count, which follow from marginal and pairwise error statistics, so it can be computed without running a decoder. Under the Gaussian-margin reference model, for example, these moments require only univariate and bivariate Gaussian probabilities, and the resulting bound holds conditional on that pairwise model. Because the error count is an integer between 0 and N, the same two moments also give sharp lower and upper bounds through a small linear program. The upper bound never exceeds Cantelli. On clean CIFAR-10, conditional on the measured moments, the interval for hard-decoder failure is 2.8–50.9%, with 27.2% observed failure; Cantelli gives 63.0%. Under the tested corruptions, the lower bounds certify failure rates of at least 41.9–57.4%, while the upper bounds remain near 100%. The moments therefore show that hard decoding is unreliable under these corruptions without evaluating the decoder. When decoded outcomes are available, the failure rate can be measured directly. Section 3.3 now states this, and a new table in Appendix F reports the per-condition results. The guarantee applies to hard bounded-distance decoding; we evaluate BP and soft readout empirically.
Reviewer_CqxfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Coded Neural Networks (CodedNN), which encode labels into redundant binary outputs and train a shared network to predict them. In multilabel classification, the original labels are kept, while a low-density parity-check code adds extra parity bits as further training targets and can be exploited by a belief-propagation decoder. For single-label classification, each class gets represented by redundant binary codewords and prediction picks the closest one. As all outputs come from the same backbone, the errors are strongly dependent, so the authors derive an error bound that does not assume independent bit errors. Across 13 multilabel classification benchmarks, CodedNN is able to improve the micro-F1 over a binary-cross-entropy predictor by 1.5 points on average. Furthermore, on CIFAR-10-C and under feature noise on ImageNet, the coded models are also more robust than softmax.

Critical strengths
  1. The experimental design is careful, understandable and transparent. The authors compare against a head with the same output size but no parity structure, run decoder ablations over a high number of seeds with confidence intervals, and report the results of all seeds. Furthermore, they also report transparently the results that limit their method, for example the negative effect on Enron. This provides the reader with an honest view on the proposed method.
  2. In the paper, the authors show that the classical coding assumption of independent errors does not hold for neural networks with a shared backbone. On Delicious, the output errors have a mean correlation of 0.45 and most of the variance lies in very few directions. The CIFAR-10 experiment also shows that two codebooks with the same length and minimum distance lead to different robustness levels. Those findings are particularly useful for anyone who wants to apply error-correcting codes to neural outputs.
Critical weaknesses
  1. It is not really clear that the gains come from the coding itself, and the main numbers are presented more positively than the tables support. When CodedNN is compared to the stronger of the two baselines on each dataset, the baseline is actually better on average. The reported average gain of 1.51 points over BCE is driven mainly by one dataset and falls to under one point without this dataset. In addition, CodedNN uses an extra label loss with class imbalance weights, and the paper does not say whether the baselines get the same weighting. In case the baselines do not get this weighting, this could explain the gains on some of the datasets, such as Enron and Birds. The capacity-matched control supports this concern. On Enron, CodedNN beats the matched control by only 1.87 points, much less than its 9.39-point gain over BCE. On CIFAR-10-C, the confidence interval for the BCH code includes zero, so it is not clearly better than softmax, and the best result origns from a nonlinear codebook.
  2. The theory and the decoder experiments are only slightly connected to the results. The Cantelli bound is never checked on real data. It also does not cover the two ways the paper actually makes predictions, raw readout and BP. The BP results do not look like error correction either. The bit error rate barely changes, and on Enron BP raises micro-F1 while macro-F1 and mAP both decrease. One could assume that BP shifts the decision trade-off instead of correcting errors.
Questions for authors

Clarification:

  1. Do the BCE baseline and the overcomplete control use the same imbalance-weighted label loss as CodedNN?
  2. Can you evaluate the Cantelli bound on real data and compare it to observed failure rates?
  3. On Enron, BP increases micro-F1 but lowers macro-F1 and mAP, while the bit error rate barely changes. What does BP actually change here?
  4. How many seeds were used for ImageNet, and were both models trained for the same number of epochs?
  5. Why does ASL perform about 9 points worse than BCE on Enron? How was ASL tuned?

Suggestions for future work:

  1. Evaluation under real distribution shifts such as ImageNet-C instead of synthetic feature noise.
5 comments

Thank you for the detailed comparisons. We now report the stronger-baseline comparison, clarify the losses, and add validation-tuned ASL results. We also checked the Cantelli bound on a hard decoder, reran BP, and documented the ImageNet training budgets.

  • Baseline weighting and claims (§§4–5, Tables 1–3). The closest source revision matching the archived runs shares the capped positive-class weights between the BCE baseline and CodedNN's weighted label term. The overcomplete control uses that weighted term with scale λ = K/(2N) and omits the full-codeword BCE term. The revision states the 15% validation holdout for official-split datasets and documents AdamW, cosine annealing, and norm-1 gradient clipping from the matching source. Across 13 datasets, the mean advantage over BCE is 1.51 points, about 0.85 excluding Enron. The stronger of BCE and LSEP per dataset leads CodedNN by about 0.21 points on average. Enron's +9.39 points over BCE in Table 1 contrasts with +1.87 over the overcomplete control in the separate 24-seed study. We narrowed the headline and added the five-arm comparison in Table 3.
  • ASL (Appendix D). The archived runs used γ₊ = 0, γ₋ = 4, and clipping 0.05. We now select γ₋ from {0, 1, 2, 4} on validation data and compare against direct-training CodedNN over ten seeds. The selected values for Emotions, Birds, and Enron are 2, 2, and 0. Tuned ASL leads on Emotions, while CodedNN leads on Birds and Enron. On Enron, tuned ASL reaches 0.3258 versus 0.4132 for CodedNN. Validation favors γ₋ = 0, showing sensitivity to this setting. The original ASL–BCE gap remains unexplained by this experiment alone. The original CodedNN column used end-to-end training and covariance fine-tuning.
  • CIFAR-10-C and Cantelli (Table 5, §3.3, Appendix F). The BCH gain over softmax is +0.61 points with a 95% interval of [−0.58, +1.94]. The nonlinear codebook gains +1.67 points. In the additional five-seed hard-decoder check, clean CIFAR-10 error is 27.16%, with a mean plug-in Cantelli bound of 63.02%. Under the three tested severity-3 corruptions, the mean bit-error count exceeds the correction threshold of two and the bound is trivial. The guarantee is for hard bounded-distance decoding. The raw multilabel readout, BP, and soft codeword scoring are evaluated empirically.
  • BP (§6.2, Tables 2 and 4). After fixing the check-node update, the raw and overcomplete metrics reproduce exactly. On clean Enron, micro-F1 rises from 0.4473 to 0.5575, while macro-F1 falls from 0.1961 to 0.1494 and mAP from 0.2134 to 0.1968. At σ = 2, Enron gains 1.35 micro-F1 points (95% paired interval [0.72, 2.03]) and BER falls by 0.47 points. The mAP drop shows that BP also changes score rankings. On Scene, the corrected BP-minus-raw BER change at σ = 2 moved from +0.07 to −0.01 points, and clean BP mAP changed from 0.792 to 0.764 (Tables 2 and 4).
  • ImageNet (§7, Figure 2, Appendix B). We used one training seed (42) per method and one noise draw per σ. Both scheduled five head-only epochs followed by up to 20 full fine-tuning epochs. Softmax completed 11 epochs with patience 5, and CodedNN completed 18 with patience 7. Both monitored validation accuracy, and selection and reporting used the same validation split. At σ = 3, CodedNN reaches 60.13% versus 47.94% for softmax, a 12.19-point gap. The revision treats this as a descriptive comparison under controlled feature noise. We agree that ImageNet-C would be a useful test of robustness to image corruptions. Figure 2 measures sensitivity to controlled feature perturbations.

I thank the authors for their response. I suggest that the authors mark the changes in the revision with colored text.

I appreciate the new experiment answers my weighting question, but it also confirms LDPC shows no consistent advantage over repetition. Changing only the loss composition shifts the score by up to 1.09 points, which is comparable to the coded-target effects. This suggests that most of the gain comes from auxiliary supervision and the loss setup. Could the authors clarify why CodedNN on Enron reaches different scores in different tables (Table 1, 9 and 14), given the small std of 0.009?

Thank you for the follow-up. We have uploaded the revised PDF with additions marked in blue.

We agree that LDPC shows no consistent absolute micro-F1 advantage over repetition in the matched experiment. Repetition is itself a simple code, and this comparison tests whether LDPC parity adds value beyond repeating the labels. Relative to the width- and loss-matched head without auxiliary targets (arm B), repetition improves mean micro-F1 in four of six dataset and noise settings, with a gain of 3.46 percentage points on clean Enron. LDPC improves it in three of six settings. Changing only the systematic-label loss shifts mean micro-F1 by up to 1.09 percentage points, comparable to several of these effects. The results show that redundant supervision and loss composition both matter, with effects that vary by dataset and noise level.

The three Enron scores come from separately trained experiments. Table 1 reports mean raw micro-F1 of 0.484 across seeds 1, 42, and 123 (SD 0.009, range 0.475–0.491). Table 9 reports 0.413 across ten seeds in the ASL comparison (SD 0.068, range 0.258–0.486), and Table 14 reports 0.447 across 24 seeds in the mechanism study (SD 0.055, range 0.284–0.504). The three Table 1 scores cluster near the upper end of the scores in the larger studies. Two runs in the ASL comparison and three in the mechanism study scored below 0.35, lowering their respective means. The training implementations also differed across studies, so the differences in means cannot be attributed to seed choice alone.

The Enron BCE baseline in Table 1 has an SD of 0.057 across the same three seeds. We therefore interpret the reported 9.39-percentage-point gain as an uncertain three-seed estimate. The paired comparisons within each experiment provide the basis for assessing the effects of loss composition and auxiliary targets.

Reviewer_j1zhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the strong correlations among codeword prediction errors when all coded outputs are generated from a shared neural representation. They study the correlations among neural output scores and provides an analysis of dependent bit errors. The paper further propose Coded Neural Networks (CodedNN), which jointly predict original labels and parity bits, with optional belief-propagation decoding. Experiments on multilabel classification, CIFAR-10-C, and ImageNet evaluate the effects of redundant supervision and decoding on classification accuracy and robustness.

Critical strengths

The paper studies error-correcting codes when prediction errors across code bits may be dependent. It uses the centered covariance of score residuals to examine dependence among outputs from the shared backbone, and separately develops a dependence-aware analysis of the total number of bit errors.

Critical weaknesses

The main MLC results do not fully isolate the effect of parity coding. CodedNN combines codeword BCE with an additional class-weighted BCE loss on the systematic labels. The reported gains over BCE cannot be clearly attributed to parity coding or the loss. Also, the robustness benefit of parity-structured supervision is inconsistent across datasets. In the output-dimension-matched study, the clean-adjusted effect is small on Emotions and Scene, and even negative on Enron. This makes it difficult to conclude that parity supervision reliably improves robustness.

The MLC experiments use a fixed rate- 1 / 3 1/3 LDPC code with column weight 3, without comparing alternative code designs under the same neural setup. Since code rate and parity structure affect both target difficulty and decoding, it is unclear whether the MLC results generalize beyond this particular choice.

Questions for authors
  1. Could the authors clarify the targets and loss used for the output-dimension-matched overcomplete control? Since CodedNN’s codeword BCE includes parity targets, removing parity structure also changes the auxiliary supervision. How is the control’s loss adapted and scaled, and how does this comparison distinguish the effect of parity structure from the effect of adding extra supervised outputs?

  2. What motivated the choice of a rate- 1 / 3 1/3, column-weight-3 LDPC code for the MLC experiments? Since the parity structure affects the learning targets and decoding, it would help to know whether the results are specific to this configuration or are expected to hold for other code rates or parity structures

1 comment

Thank you for highlighting the ambiguity in the original control. We now specify its loss and report a five-arm experiment that matches systematic supervision and varies the auxiliary targets within each dataset. The results show that the contribution of parity depends on the dataset and noise level.

  • Original control (§4, Table 2). The overcomplete head has the same output width as CodedNN. Only the first C outputs receive weighted label BCE, scaled by λ = K/(2N). The remaining outputs receive no direct loss. CodedNN also receives unweighted BCE over the full codeword, including the systematic labels, padding, and parity. The original comparison therefore changes both the systematic objective and auxiliary supervision.
  • Matched controls (§6.1, Table 3). Across ten paired seeds per dataset, LDPC versus the width- and systematic-loss-matched control (D−B) gives +0.77 points on clean Emotions. At σ = 2, the changes are +0.29, +0.39, and −1.61 points on Emotions, Scene, and Enron. Changing only the systematic-loss composition (A−A′) shifts micro-F1 by up to 1.09 points. LDPC versus repetition (D−C) shows mixed absolute gains. On Enron the +3.30-point clean-adjusted difference combines a −3.02-point clean gap and a +0.27-point corrupted gap. We narrowed the robustness claim accordingly.
  • Code choice (§4, Appendix E). The rate-1/3 construction fixes the output budget, and sparse checks keep BP practical as codewords grow. The sensitivity sweep reports trainer-selected and fixed column weights 3 and 6 separately for each dataset. On Enron, weight 6 improves clean micro-F1 while reducing corrupted micro-F1 relative to the trainer-selected setting. On Emotions, the shortened RPTU design and the automatically generated matched-loss control give similar results. These experiments leave rate sensitivity open.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes in the Revised Manuscript

We thank the reviewers for their constructive feedback. We have uploaded a revised PDF. The main text uses the additional page to explain the matched controls and BP results, and the changes are summarized below. The revised experiments show when redundant outputs help and which metrics improve. The five-arm controls distinguish loss composition from auxiliary supervision, and the CIFAR-10-C comparison shows that codebooks with the same length and minimum distance can yield different corruption accuracy.

  • Matched controls (§6.1, Table 3). We added a ten-seed, five-arm experiment on Emotions, Scene, and Enron. At matched width and systematic loss, the effects of LDPC targets vary by dataset. LDPC also shows no consistent absolute micro-F1 advantage over repetition. Appendix E reports the LDPC design sensitivity checks.
  • Multilabel baselines (§§4–5, Appendix D). We clarify the baseline losses and training protocol and report comparisons to both weighted BCE and the stronger baseline per dataset. The validation-tuned ASL comparison favors ASL on Emotions and direct-training CodedNN on Birds and Enron.
  • Decoder audit (§6.2, Tables 2 and 4). We fixed the BP check-node update and reran the 24-seed comparisons. Raw and overcomplete metrics reproduced exactly. BP improves Enron micro-F1 while lowering macro-F1 and mAP.
  • Dependence and scope (§§3.2–3.3, Appendices D and F). We identify the original Delicious covariance as a test-split diagnostic and replace the Gaussianity plot with a validation diagnostic from the trained checkpoint. The new CIFAR-10 hard-decoder check shows a loose clean-data Cantelli bound and a trivial bound under the tested corruptions.
  • Single-label results (§7, Table 5 and Figure 2). We distinguish the nonlinear codebook result from the BCH result on CIFAR-10-C. The feature-noise figure identifies the auxiliary CIFAR-10 runs and the descriptive ImageNet comparison, including the unequal training budgets and shared checkpoint-selection and reporting split.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.