acceptodds
Under review as a conference paper at ICLR 2027

Parameter-Efficient Error Correction Code Transformers

Abstract

Error correction coding is a cornerstone of reliable communication systems, ensuring data integrity over noisy channels. The Error Correction Code Transformer (ECCT) and its successors have demonstrated that Transformer-based neural decoders can outperform classical decoding methods for short and medium linear block codes. However, existing Transformer-based decoders suffer from high parameter counts and computational costs, both of which scale linearly with model depth, constraining their deployment scenarios and performance potential. In this work, we propose two architectures to tackle the parameter efficiency challenge. First, we introduce the Shared Error Correction Code Transformer (SECCT), which employs cross-layer parameter sharing, effectively collapsing the model to a single Transformer layer that is applied repeatedly. By preserving decoding depth through iteration rather than stacked distinct layers, SECCT improves the bit error rate (BER) over the published ECCT baseline on all 39 code–signal-to-noise ratio (SNR) settings we evaluate while using less than 20% of the parameters of that baseline. The price is only a modest increase in computational cost due to additional forward passes. This provides a viable solution with performance guarantees for memory-constrained edge deployment scenarios. Second, we propose the Hybrid Error Correction Code Transformer (HECCT), which employs Grouped-Query Attention (GQA) and a bottleneck feed-forward structure to simplify each Transformer layer. HECCT achieves lower parameter count and computational cost than the published ECCT baseline, while delivering better BER performance, establishing a new Pareto-optimal frontier in the accuracy–efficiency trade-off. Extensive experiments on 13 codes from five families validate the effectiveness of both architectures, which provide a reliable and efficient solution for next-generation communication systems.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
parameter efficiency, transformer architectures, cross-layer parameter sharing, error correction codes, channel decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vZ7XOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work targets the parameter cost of the Error Correction Code Transformer (ECCT) and its foundation-model successor (FECCT). The authors pursue two directions: reusing Transformer layers across depth, and Grouped-Query Attention (GQA) with a bottleneck FFN, corresponding to SECCT and HECCT respectively.

In the reported experiments, SECCT achieves better performance than ECCT while storing only about 18% of its parameters, and HECCT uses fewer parameters than ECCT while implementing a deeper model (12 layers vs. 6). SECCT and HECCT deliver comparable results.

Critical strengths

This work answers a question: how an ECCT-style model can be simplified, both along depth (layer reuse) and within each layer (simplified attention and FFN architecture).

Critical weaknesses
  • Missing comparisons with recent SOTA methods. While ECCT can be regarded as an early work on parity-check-matrix-masked Transformers, several subsequent methods, including CrossMPT, EfficientMPT, and SBECCT, are neither sufficiently discussed nor included as baselines. This makes the related-work section incomplete and, more importantly, makes it difficult to assess whether the proposed methods actually improve upon the current state of the art. Based on the reported results, it is also unclear that the proposed approaches consistently outperform these more recent models.
  • The methodological novelty appears limited. In particular, given the existence of EfficientMPT and other closely related architectures, the proposed contribution seems largely incremental. The paper should provide a much clearer comparison with these prior methods and explicitly identify which architectural or algorithmic components are genuinely novel, rather than variations of existing designs.
  • The connection between SECCT and HECCT is weak. The two proposed architectures appear to be motivated and designed largely independently, and the manuscript does not provide a sufficiently strong conceptual or technical connection between them. In the current presentation, combining them into a single paper feels somewhat artificial; each method could arguably constitute an independent work. A clearer unifying perspective is needed to justify presenting both methods together.
  • The paper lacks an experiment that directly connects SECCT and HECCT. A natural experiment would be to construct a hybrid SECCT+HECCT model, for example by reusing or sharing HECCT layers/components within SECCT. Such an experiment would help determine whether the two proposed approaches are complementary and would provide stronger empirical support for treating them as parts of a unified framework.
Questions for authors

Minor issues:

  • Please use \citet and \citep appropriately according to their grammatical roles in the sentence. The manuscript currently appears to use \citet for nearly all citations. This makes the prose awkward and reduces the overall readability and polish of the paper.
7 comments
AuthorsOpenReview ↗

Response to Review vZ7X (1/3, W1&Minor)

W1|Missing comparison with the recent state of the art

Reviewer's comment

Missing comparisons with recent SOTA methods. While ECCT can be regarded as an early work on parity-check-matrix-masked Transformers, several subsequent methods, including CrossMPT, EfficientMPT, and SBECCT, are neither sufficiently discussed nor included as baselines. This makes the related-work section incomplete and, more importantly, makes it difficult to assess whether the proposed methods actually improve upon the current state of the art. Based on the reported results, it is also unclear that the proposed approaches consistently outperform these more recent models.

We accept this criticism. The failure to include these three works in the original submission was an oversight rather than a deliberate choice.

The gap is now closed: a new paragraph in Section 2 (Related Work) discusses CrossMPT, EfficientMPT and AECCT; Table 1 in Section 4.1 gains three baseline columns covering all 13 codes at 4/5/6 dB; and a new Table 2 gives the parameters and MACs of these methods.

For space, the table below gives only representative codes and an ellipsis marks the rest:

CodeSNRECCTFECCTAECCTCrossMPTEfficientMPTSECCTHECCT
LDPC(121,70)46.19-6.387.066.967.107.06
59.89-10.1011.3911.3111.5611.51
615.58-16.0117.5216.9717.3517.33
LDPC(121,80)47.07-7.277.997.958.128.08
510.96-11.5012.7512.4012.8112.67
616.25-16.9018.1517.9018.3818.32
⋮⋮⋮⋮⋮⋮⋮⋮⋮

and the corresponding parameters and MACs:

CodeQuantityECCTCrossMPTEfficientMPTSECCTHECCT
BCH(31,16)Parameters1.20M (100%)1.20M (100%)1.09M (91%)0.21M (17%)0.71M (59%)
MACs59.0M (100%)57.1M (97%)52.1M (88%)118.0M (200%)39.4M (67%)
BCH(63,45)Parameters1.21M (100%)1.21M (100%)1.09M (91%)0.21M (18%)0.72M (59%)
MACs105.9M (100%)99.3M (94%)91.8M (87%)211.8M (200%)76.4M (72%)
⋮⋮⋮⋮⋮⋮⋮

The results indicate that the proposed schemes reach a performance comparable to the current state of the art, with savings in both parameters and computation. Both of our designs surpass the earlier decoders, namely the BP-based methods together with ECCT and FECCT, on all 39 code–SNR settings; against the recent AECCT, CrossMPT and EfficientMPT, they reach a similar performance and exceed them in a number of cases. In parameters, SECCT stores the fewest on every code, 82–83% below ECCT and 80–81% below EfficientMPT, at the cost of a MAC count twice that of ECCT; HECCT stores 40–41% fewer parameters than ECCT and 34–35% fewer than EfficientMPT, and its MACs are the lowest of all compared methods on every code. The proposed methods therefore reach the state of the art and lead in the parameter budget.

Changes made

  • Section 2, Related Work: a new paragraph introducing the mechanisms of CrossMPT, EfficientMPT and AECCT
  • Section 4.1, Table 1: three new baseline columns (AECCT, CrossMPT, EfficientMPT) covering all 13 codes at 4/5/6 dB
  • Section 4.2, Table 2: a new table of parameters and MACs with percentages relative to ECCT

Minor|Inappropriate use of \citet and \citep

Reviewer's comment

Please use \citet and \citep appropriately according to their grammatical roles in the sentence. The manuscript currently appears to use \citet for nearly all citations. This makes the prose awkward and reduces the overall readability and polish of the paper.

We have distinguished the two citation forms according to their grammatical roles: \citet where the author name forms part of the sentence, and \citep for parenthetical citations. All citations in the body text have been adjusted accordingly.

Changes made

  • Throughout the paper: citation format distinguished by grammatical role, \citet where the author name is part of the sentence, \citep for parenthetical citations
AuthorsOpenReview ↗

Response to Review vZ7X (2/3, W2)

W2|Limited methodological novelty, incremental with respect to EfficientMPT

Reviewer's comment

The methodological novelty appears limited. In particular, given the existence of EfficientMPT and other closely related architectures, the proposed contribution seems largely incremental. The paper should provide a much clearer comparison with these prior methods and explicitly identify which architectural or algorithmic components are genuinely novel, rather than variations of existing designs.

Regarding the difference from these methods, we consider the contributions to be unlike one another: each of them modifies a different component, and they therefore cannot substitute for each other. For ease of comparison, the table below lists the core contribution of each method and the component it modifies; the same comparison is given in Section E of the appendix.

MethodCore contributionComponent modified
CrossMPTUpdates the magnitude and syndrome embeddings separately through masked cross-attention, so that the two kinds of information are iterated in their own channelsattention form (self → cross)
EfficientMPTEEC attention: keeps only the query–key interaction, replaces the per-position query with a global query vector, and removes the value projectionhow attention is computed in each layer
AECCTTernary weight quantization (AAP), head-partitioned self-attention (HPSA), spectral positional encoding (SPE)weight precision and attention structure
SECCTReplaces the independently stored parameters of each layer with a single shared parameter set applied repeatedly along the depth: every layer performs the same mapping, so the number of iterations is decoupled from parameter storage, and an arbitrarily deep decoding network can be emulated at constant storageparameter storage between layers
HECCTCompresses two sub-structures within a layer at the same time: on the attention side it narrows the key and value projections and shares one set of key–value projections across several query heads; on the feed-forward side it compresses the hidden width, so that the intra-layer nonlinearity operates in a lower-dimensional subspacekey–value projection width and feed-forward hidden width within a layer

It can be seen that CrossMPT, EfficientMPT and AECCT all act on how computation is performed within a layer; their number of layers and embedding dimension are unchanged, so their storage still grows linearly with depth. In particular, the parameter count of SECCT does not grow with depth, which differs markedly from the optimisation direction of the prior papers: it does not lower the cost of each layer but removes the per-layer storage itself.

In summary, the difference between this paper and prior algorithms lies in two respects: the optimisation direction and the optimisation object.

For SECCT the direction differs from prior work: prior methods optimise the resource usage within a layer, whereas SECCT addresses the resource usage between layers, so its parameter count does not grow with depth.

For HECCT the direction is the same as in prior work, all acting on the resources within a layer, but the object differs: prior methods compress the way attention is performed, the weight precision or the feed-forward structure, whereas HECCT compresses the projection width between keys and queries and the hidden width of the feed-forward.

Changes made

  • Section 2, Related Work: a new paragraph introducing the mechanisms of CrossMPT, EfficientMPT and AECCT
  • Section 4.2, Table 2: a new table of parameters and MACs, allowing the storage and computation of the three compression schemes to be compared item by item
  • Section 3, SECCT and HECCT subsections: state that the two forms of compression act between and within layers, respectively
  • Section E of the appendix: a new section containing a comparison of the component each method modifies and the resource it reduces
AuthorsOpenReview ↗

Response to Review vZ7X (3/3, W3&W4)

W3|The connection between SECCT and HECCT is weak

Reviewer's comment

The connection between SECCT and HECCT is weak. The two proposed architectures appear to be motivated and designed largely independently, and the manuscript does not provide a sufficiently strong conceptual or technical connection between them. In the current presentation, combining them into a single paper feels somewhat artificial; each method could arguably constitute an independent work. A clearer unifying perspective is needed to justify presenting both methods together.

We accept this. The presentation in the original submission did not make the relation between the two clear. Section E of the appendix discusses this point specifically, and the content is summarised below.

SECCT and HECCT represent two different optimisation directions: one addresses the resources between layers, the other the resources within a layer.

SECCT addresses the resources between layers. It replaces the independently stored parameters of each layer with a single shared parameter set applied repeatedly along the depth; every layer performs the same mapping, so parameter storage is decoupled from the number of iterations, and an arbitrarily deep decoding network can be emulated at constant storage. What it reduces is the parameter storage between layers.

HECCT addresses the resources within a layer. It compresses two sub-structures within a layer at the same time: on the attention side it narrows the key and value projections and shares one set of key–value projections across several query heads; on the feed-forward side it compresses the hidden width, so that the intra-layer nonlinearity operates in a lower-dimensional subspace. What it reduces is the resource occupied by each sub-structure within a layer.

These two optimisation directions are orthogonal, and we present them separately in order to provide a solution for each direction:

  • extreme parameter compression with a constant parameter count, at the cost of higher computation;
  • a simultaneous reduction of both parameters and computation.

This better demonstrates the effect of the two optimisation directions, and is of practical value for different deployment scenarios.

Changes made

  • Section E of the appendix: a new section explaining that the two architectures act on different resources and are orthogonal, together with a comparison against the prior methods
  • Section 3: the SECCT and HECCT subsections state that the compression acts between and within layers, respectively
  • Section 4.2, Table 2: the storage and computation effects of the two forms of compression are placed side by side in one table

W4|Missing experiment directly connecting the two

Reviewer's comment

The paper lacks an experiment that directly connects SECCT and HECCT. A natural experiment would be to construct a hybrid SECCT+HECCT model, for example by reusing or sharing HECCT layers/components within SECCT. Such an experiment would help determine whether the two proposed approaches are complementary and would provide stronger empirical support for treating them as parts of a unified framework.

Our intention was to present the two schemes separately, as solutions for different application scenarios. For example, the constant-level parameter storage of SECCT suits edge devices with severely limited storage and memory, where performance is traded for by increasing the number of iterations; if the storage budget is somewhat more generous, HECCT can be used to further reduce the computational cost and improve performance.

We accept the reviewer's point that an experiment connecting the two is necessary, since otherwise it causes confusion for readers. We have made every effort to run the simulations related to this response, and will revise the paper and add a comment at the first moment the results are available.

Reviewer_QQ5zOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates parameter-efficient Transformer architectures for neural error-correction decoding. The authors first analyze redundancy in a trained ECCT model, including cross-layer and cross-head CKA, the effective rank of key/value projections, and the low effective dimensionality of intermediate FFN activations. Based on these observations, they propose two ECCT variants: SECCT, which replaces independently parameterized Transformer layers with a single shared layer applied recurrently, and HECCT, which combines Grouped-Query Attention (GQA) with a bottleneck FFN.

The experimental results demonstrate substantial parameter savings. For example, on Polar(64,32), SECCT uses only 217K parameters compared with 1.21M for the published six-layer ECCT baseline, although its MACs double. HECCT reduces both parameter count and MACs while achieving better BER than the same baseline.

I recognize the practical importance of parameter efficiency in neural decoding. However, I have two major concerns regarding this paper:

  1. The current performance comparisons are substantially confounded by increased decoding depth, since the main ECCT reference uses six layers whereas SECCT uses 12 shared iterations and HECCT uses 12 independent layers.

  2. The core techniques employed by SECCT and HECCT are largely adopted from existing work on parameter-efficient Transformers, resulting in limited methodological novelty.

Critical strengths
  1. Practical relevance. The parameter count and computational cost of Transformer-based neural decoders are important limitations for practical deployment.

  2. Meaningful redundancy analysis. The CKA and effective-rank analyses provide empirical evidence of redundancy in ECCT and reasonable motivation for developing more compact architectures.

Critical weaknesses
  1. Table 1, Figure 6, and the experimental settings reveal an important issue: the main ECCT baseline uses six layers, whereas SECCT uses 12 shared iterations and HECCT uses 12 independent layers. Increasing network depth can improve decoding performance, as also demonstrated by the paper's own Figure 6(b). Moreover, Figure 6(a) compares the parameter counts of equally deep shared and unshared models but does not report the decoding performance of the unshared models. Therefore, the current experiments do not clearly distinguish the performance gains from increased depth and those attributable to the proposed architectures.

  2. I acknowledge that HECCT achieves lower parameter count and MACs than the six-layer ECCT baseline despite using 12 layers. However, given the importance of network depth, the authors should compare HECCT with a conventional 12-layer ECCT to determine whether it can retain comparable or better decoding performance with fewer parameters and lower computational cost. In addition, HECCT is trained for 1,500 epochs, whereas the six-layer ECCT baseline is trained for 1,000 epochs.

  3. I want to know why the author increased the network depth instead of keeping it at 6 layers.

  4. The main architectural components used in this paper are already well-established in the Transformer literature. SECCT adopts cross-layer parameter sharing, previously explored in ALBERT and Universal Transformers. HECCT combines standard GQA with a reduced-width FFN, both of which are existing parameter-efficient Transformer techniques.

  5. I believe comparisons with AECCT and EfficientMPT would help better establish the efficiency advantages of the proposed methods.

Questions for authors
  1. The paper should more clearly distinguish the performance improvements resulting from increased decoding depth from those attributable to parameter sharing and architectural compression.

  2. Beyond applying established Transformer parameter-sharing and compression techniques to ECCT, what do the authors consider the main methodological innovation specific to neural error-correction decoding?

  3. Additional questions and concerns are discussed in the Major Weaknesses section above.

6 comments
AuthorsOpenReview ↗

Response to QQ5z (1/3, W2&W6&W7&Q2)

W2&W6&W7&Q2|Methodological novelty and comparison: borrowed components, our contribution, and the SOTA comparison

Reviewer's comment

The core techniques employed by SECCT and HECCT are largely adopted from existing work on parameter-efficient Transformers, resulting in limited methodological novelty.

The main architectural components used in this paper are already well-established in the Transformer literature. SECCT adopts cross-layer parameter sharing, previously explored in ALBERT and Universal Transformers. HECCT combines standard GQA with a reduced-width FFN, both of which are existing parameter-efficient Transformer techniques.

I believe comparisons with AECCT and EfficientMPT would help better establish the efficiency advantages of the proposed methods.

Beyond applying established Transformer parameter-sharing and compression techniques to ECCT, what do the authors consider the main methodological innovation specific to neural error-correction decoding?

We agree that these components are well-established in the Transformer literature. This paper does not propose new constructions in the basic architecture or mechanisms of the general Transformer; what we do is borrow some effective techniques from the Transformer literature (such as cross-layer parameter sharing, GQA and the bottleneck structure) and use them to improve the ECCT architecture, obtaining lower complexity and higher accuracy.

Such improvements are not unfamiliar in this area. For example, the masked cross-attention adopted by CrossMPT borrows the cross-attention mechanism used for cross-stream interaction in the Transformer literature. Also, we believe that improving the ECCT architecture still holds potential, and more precise and code-specific improvements to ECCT will be our future work.

To avoid giving readers the impression that these components are new mechanisms of this paper, we have simplified the corresponding descriptions in Related Work and added introductions to the SOTA algorithms (AECCT, CrossMPT, EfficientMPT and others).

We have also added comparisons with CrossMPT, EfficientMPT and AECCT. The results indicate that the proposed schemes reach a performance comparable to the current state of the art, with savings in both parameters and computation. For space, the tables below give only representative codes and an ellipsis marks the rest:

CodeSNRECCTFECCTAECCTCrossMPTEfficientMPTSECCTHECCT
LDPC(121,80)47.07-7.277.997.958.128.08
510.96-11.5012.7512.4012.8112.67
616.25-16.9018.1517.9018.3818.32
⋮⋮⋮⋮⋮⋮⋮⋮⋮

and the corresponding parameters and MACs:

CodeQuantityECCTCrossMPTEfficientMPTSECCTHECCT
BCH(63,45)Parameters1.21M (100%)1.21M (100%)1.09M (91%)0.21M (18%)0.72M (59%)
MACs105.9M (100%)99.3M (94%)91.8M (87%)211.8M (200%)76.4M (72%)
⋮⋮⋮⋮⋮⋮⋮

Both of our designs surpass the earlier decoders, namely the BP-based methods together with ECCT and FECCT, on all 39 code–SNR settings; against the recent AECCT, CrossMPT and EfficientMPT, they reach a similar performance and exceed them in a number of cases. In parameters, SECCT stores the fewest on every code, 82–83% below ECCT and 80–81% below EfficientMPT, at the cost of a MAC count twice that of ECCT; HECCT stores 40–41% fewer parameters than ECCT and 34–35% fewer than EfficientMPT, and its MACs are the lowest of all compared methods on every code. The proposed methods therefore reach the state of the art and lead in the parameter budget.

In summary, the contribution of this paper lies not in introducing new Transformer components, but in compressing resources effectively along two dimensions of ECCT: the depth direction (layer sharing, which decouples storage from the number of iterations) and the intra-layer direction (width compression of attention and FFN).

The two directions share the same goal of making each layer cheaper. Note that parameter compression reduces resource use at the cost of accuracy, whereas increasing the number of layers raises accuracy at the cost of resources. When compression can be achieved at a small accuracy cost, the resources saved can be spent on additional layers, so that both lower resource use and higher accuracy are obtained within a given resource budget.

Changes made

  • Section 2, Related Work: a new paragraph introducing the mechanisms of CrossMPT, EfficientMPT and AECCT
  • Section 4.1, Table 1: three new baseline columns (AECCT, CrossMPT, EfficientMPT) covering all 13 codes at 4/5/6 dB
  • Section 4.2, Table 2: a new table of parameters and MACs with percentages relative to ECCT
AuthorsOpenReview ↗

Response to QQ5z (2/3, W1&W3)

W1|Performance gains confounded by increased depth

Reviewer's comment

Table 1, Figure 6, and the experimental settings reveal an important issue: the main ECCT baseline uses six layers, whereas SECCT uses 12 shared iterations and HECCT uses 12 independent layers. Increasing network depth can improve decoding performance, as also demonstrated by the paper's own Figure 6(b). Moreover, Figure 6(a) compares the parameter counts of equally deep shared and unshared models but does not report the decoding performance of the unshared models. Therefore, the current experiments do not clearly distinguish the performance gains from increased depth and those attributable to the proposed architectures.

We agree that depth affects performance, but this is precisely the design premise of SECCT; we are not pursuing accuracy at equal depth. We have added the performance statistics of the 12-layer ECCT to the paper:

MethodParametersMACs4 dB5 dB6 dB
ECCT (L=6L=6)1.21M127.7M6.999.4412.32
SECCT (S1, L=12L=12)217K255.4M7.289.6513.04
HECCT (S2, L=12L=12)0.72M95.0M7.469.8313.20
ECCT (L=12L=12)2.40M254.8M7.5010.3013.66

Some explanation of this issue follows.

SECCT replaces per-layer storage with a single iterated layer: as long as one parameter set is stored, increasing the number of iterations does not increase storage. Depth in SECCT is therefore a degree of freedom that can be adjusted at deployment time; we use it to trade for accuracy rather than treating it as a variable to be controlled, which has practical value on edge devices with limited storage or memory. The ablation study shows that this degree of freedom is indeed effective.

The cost of the depth-matched reference is listed as well. The 12-layer unshared ECCT attains the highest accuracy of the four, but at the same time carries the complexity cost in both storage and computation: 2.40M parameters, 11.0×11.0\times those of SECCT and 3.3×3.3\times those of HECCT, and 254.8M MACs per codeword, 2.7×2.7\times those of HECCT. An ECCT with the same parameter count as the 12-layer HECCT or SECCT would necessarily have fewer than six layers, and would therefore perform worse than the six-layer ECCT baseline.

This shows that the benefits of the two act in opposite directions: the compression of our methods targets resources at the cost of accuracy, whereas added depth targets accuracy at the cost of resources. When the accuracy cost of compression is small, the resources saved can be spent on additional depth, so that both lower resource use and higher accuracy are obtained.

Changes made

  • Section 4.2, Table 3: a new row for the 12-layer unshared ECCT, giving its parameters, MACs per codeword and −ln⁡(BER)-\ln(\text{BER}) at 4/5/6 dB, as the depth-matched reference for SECCT and HECCT (both apply twelve iterations)
  • Section 4.2, body text: a new paragraph explaining the accuracy and the cost of this depth-matched reference

W3|HECCT should be compared with a conventional 12-layer ECCT

Reviewer's comment

I acknowledge that HECCT achieves lower parameter count and MACs than the six-layer ECCT baseline despite using 12 layers. However, given the importance of network depth, the authors should compare HECCT with a conventional 12-layer ECCT to determine whether it can retain comparable or better decoding performance with fewer parameters and lower computational cost.

This has been done. A 12-layer ECCT has been trained with the same setting as HECCT, and its performance and cost are now reported in the paper. Because SECCT and HECCT both apply twelve iterations, Table 3 lists, besides the two, a 12-layer unshared ECCT as the depth-matched reference.

MethodParametersMACs6 dB
HECCT (S2, L=12L=12)0.72M95.0M13.20
ECCT (L=12L=12)2.40M254.8M13.66

Of the three, the 12-layer ECCT attains the highest accuracy, 13.66 at 6 dB, 0.46 above HECCT; however, its parameter count is 2.40M, 3.3×3.3\times that of HECCT, and its MACs per codeword are 254.8M, 2.7×2.7\times those of HECCT. HECCT is therefore not more accurate at equal depth; it approaches the accuracy of a depth-matched model at a lower resource cost.

Changes made

  • Section 4.2, Table 3: a row for the 12-layer unshared ECCT, giving both its performance and its cost
  • Section 4.2, body text: a new note that the 12-layer ECCT attains the highest accuracy but at a higher parameter and MAC cost
AuthorsOpenReview ↗

Response to QQ5z (2/3, W4&W5&Q1)

W4|HECCT trained for 1,500 epochs against a 1,000-epoch baseline

Reviewer's comment

In addition, HECCT is trained for 1,500 epochs, whereas the six-layer ECCT baseline is trained for 1,000 epochs.

First, the 1,500-epoch setting follows the established practice of ECCT and FECCT. HECCT adopts 1,500 epochs, following the 10-layer configuration of ECCT and the practice of FECCT. SECCT stores only a single shared layer, which is updated several times in every backward pass, so it follows the 1,000-epoch configuration of the six-layer models. The difference in epoch count stems from this structural distinction rather than from an advantage given to either method. Likewise, the newly added 12-layer ECCT reference also uses 1,500 epochs, identical to HECCT, so the depth-matched comparison in Table 3 is not affected by the number of training epochs.

Second, our experiments indicate that 1,000 and 1,500 epochs perform essentially the same. For five codes of HECCT (BCH(63,36), LDPC(121,70), MacKay(96,48), Polar(64,32) and CCSDS(128,64)), we took the best weight at epoch ≤ 1000 and the final weight at 1,500 epochs, and compared −ln⁡(BER)-\ln(\text{BER}) at 4 and 5 dB:

CodeSNRep=1000ep1500
BCH(63,36)51.201e-031.172e-03
LDPC(121,70)51.301e-051.304e-05
MacKay(96,48)42.474e-042.404e-04
Polar(64,32)46.485e-046.823e-04

Changes made

  • Section 4.1: the paragraph on training epochs has been rewritten to state the basis for the epoch count of HECCT and of SECCT respectively.

W5|Why increase the depth instead of keeping six layers

Reviewer's comment

I want to know why the author increased the network depth instead of keeping it at 6 layers.

Both architectures are designed to make depth inexpensive. SECCT stores parameters independently of the number of iterations, so however many iterations are run only one parameter set is stored, whereas an unshared model's storage grows linearly with depth; HECCT reduces the intra-layer cost through GQA and a Bottleneck FFN, so that even with twice as many layers its parameters and computation remain below the six-layer baseline.

Twelve layers were chosen to display the advantages of these two designs more clearly. Decoding performance improves with depth (Figure 6(b)), and at 12 layers both schemes attain better performance than the six-layer baseline with lower parameters and/or computation, which has practical value for edge devices limited in storage or computation.

Had the depth been kept at six layers, our designs would bring no performance gain and their application value would not be visible.

More fundamentally, the compression achieved by HECCT and SECCT mainly targets resources at the cost of accuracy, whereas the gain from added depth lies mainly in accuracy at the cost of resources. When HECCT and SECCT achieve efficient parameter compression at a small accuracy cost, depth can be increased to obtain both lower resource use and higher accuracy.

Consistent with W3, HECCT is not more accurate at equal depth but attains better performance at lower resource cost; the same holds for SECCT, which supports more iterations on constant storage and trades depth for accuracy.

Changes made

  • Section 3, SECCT subsection: it is stated that the parameter count is independent of the number of iterations, so that the network can be iterated arbitrarily to emulate an arbitrarily deep network
  • Section 4.3, Figure 6: the iteration-depth ablation, giving stored parameters and −ln⁡(BER)-\ln(\text{BER}) at different depths
  • Section 4.2, Table 3: parameters, MACs and accuracy of each method at the 12-layer setting

Q1|More clearly distinguishing depth gains from sharing/compression gains

Reviewer's comment

The paper should more clearly distinguish the performance improvements resulting from increased decoding depth from those attributable to parameter sharing and architectural compression.

We have added the corresponding experiments, namely a 12-layer unshared ECCT listed alongside in Table 3 as the depth-matched reference.

As stated in W5, both architectures are designed to make depth inexpensive: SECCT stores parameters independently of the number of iterations, so however many iterations are run only one parameter set is stored, whereas an unshared model's storage grows linearly with depth; HECCT reduces the intra-layer cost through GQA and a Bottleneck FFN, so that even with twice as many layers its parameters and computation remain below the six-layer baseline. Because depth becomes cheap, more layers can be spent to obtain better performance.

Changes made

  • Section 4.2, Table 3 and body text: a 12-layer unshared ECCT is added as the depth-matched reference, so that the depth gain and the compression gain can be read separately
Reviewer_7yfgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes two parameter-efficient implementations of the transformer-aided decoding scheme introduced by ECCT, called SECCT and HECCT, respectively. SECCT reuses the exact same parameters for all transformer layers, reducing the parameter count by L compared to the L-layer ECCT baseline. It also provides visible performance improvement in terms of bit error rate (BER) at all tested linear block code and channel SNR combinations. HECCT, on the other hand, introduces some degree of freedom, and employs grouped-query attention and a bottleneck feedforward structure, resulting in furher reductions in parameter count and BER.

Critical strengths
  • The paper studies an important and timely problem with significant impact. Error correction coding is a core technology for all communication systems. Reducing BER for existing codes can provide signinficant impact. Transformer-based architectures can achieveve this, although current memory and computational requirements are far higher than existing baselines. Therefore, progress in this direction is valuable.

  • The paper is well written, the idea is clear and the results are presented cleanly with ablation studies.

  • Results show visible improvements in both BER and computational complexity with significant reduction in memory requirements.

Critical weaknesses
  • There has been quite a few recent develpoments on transformer-aided decoding of error correction codes in the last few years, which are completely missing in the paper. These need to be includednot only in the Related Work seciton, but also in the Results as baselines. In particular, two highly relevant papers are: CrossMPT (ICLR 2025) and EfficientMPT (ICLR 2026). Quoting from the latter: “ EfficientMPT achieves 85% and 91% of significant memory reduction and 47% and 57% of FLOPs reduction compared to ECCT for (648,540) and (1056,880) standard LDPC codes, respectively.” Based on this, I’m not sure how the current paper compares with the state-of-the-art (SotA), and this comparison is essential for a final evaluation.

  • The novelty of the paper is limited. It applies known methods in reducing the parameter count of transformer-based models to the particular ECCT architecture. As such, it does not introduce any technical novelty. I would still appreciate this contribution if the authors can firmly show that they achieve SotA performance.

  • Results are limited to BER comparisons, which are not the main performance measure for error correction codes. Please provide BLER comparisons.

  • Results are reported only over AWGN channels. It’s pretty standard to provide similar comparisons also for fading channels.

Questions for authors

I don’t have specific questions to the authors beyond the points I have raised above. The paper is quite clear, and the applied methods are well-known and well-motivated.

  • How would the results presented in Tables 2, 3 and 4 extend to other codes? Have you run any tests?

  • Can you please explain why the computational requirement of SECCT is double that of ECCT?

  • Similarly, where is the reduction in the computational requirement of HEECT coming from?

6 comments
AuthorsOpenReview ↗

Response to Review 7yfg (1/3)

W1|Missing comparison with CrossMPT and EfficientMPT

Reviewer's comment

There has been quite a few recent developments on transformer-aided decoding of error correction codes in the last few years, which are completely missing in the paper. These need to be included not only in the Related Work section, but also in the Results as baselines. In particular, two highly relevant papers are: CrossMPT (ICLR 2025) and EfficientMPT (ICLR 2026).

We accept this. The failure to include these three works was an oversight rather than a deliberate choice.

The gap is now closed in three places: a new paragraph in Related Work discusses CrossMPT, EfficientMPT and AECCT; Table 1 gains three baseline columns covering all 13 codes at 4/5/6 dB; and Table 2 is a new table giving the parameters and the MACs of these methods.

Changes made

  • Section 2, end of Related Work: a new paragraph discussing the mechanisms of CrossMPT, EfficientMPT and AECCT and how they differ from our methods
  • Section 4.1, Table 1: three new baseline columns (AECCT, CrossMPT, EfficientMPT) covering all 13 codes at 4/5/6 dB
  • Section 4.2, Table 2: a new table of parameters and MACs listing the five methods side by side

W2|Limited novelty

Reviewer's comment

The novelty of the paper is limited. It applies known methods in reducing the parameter count of transformer-based models to the particular ECCT architecture. As such, it does not introduce any technical novelty. I would still appreciate this contribution if the authors can firmly show that they achieve SotA performance.

The idea of this paper does come from existing work on parameter-efficient Transformers. The cross-layer parameter sharing of ALBERT, grouped-query attention and the bottleneck structure were our starting points.

Building on this, we have added a comparison with CrossMPT, EfficientMPT and AECCT. The results indicate that the proposed schemes reach a performance comparable to the current state of the art, with savings in both parameters and computation. For space, the table below gives only representative codes and an ellipsis marks the rest:

CodeSNRECCTFECCTAECCTCrossMPTEfficientMPTSECCTHECCT
LDPC(121,80)47.07-7.277.997.958.128.08
510.96-11.5012.7512.4012.8112.67
616.25-16.9018.1517.9018.3818.32
⋮⋮⋮⋮⋮⋮⋮⋮⋮

and the corresponding parameters and MACs:

CodeQuantityECCTCrossMPTEfficientMPTSECCTHECCT
BCH(31,16)Parameters1.20M (100%)1.20M (100%)1.09M (91%)0.21M (17%)0.71M (59%)
MACs59.0M (100%)57.1M (97%)52.1M (88%)118.0M (200%)39.4M (67%)
⋮⋮⋮⋮⋮⋮⋮

The conclusion is as follows. Both of our designs surpass the earlier decoders, namely the BP-based methods together with ECCT and FECCT, on all 39 code–SNR settings. Against the recent state-of-the-art models AECCT, CrossMPT and EfficientMPT, they reach a similar performance and exceed them in a number of cases. Specifically, the two designs attain the best result in 23 of the 39 settings. In parameters, SECCT stores the fewest on every code, 82–83% below ECCT and 80–81% below EfficientMPT, at the cost of a MAC count twice that of ECCT; HECCT stores 40–41% fewer parameters than ECCT and 34–35% fewer than EfficientMPT, and its MACs are the lowest of all compared methods on every code, 26–33% below ECCT and 13–24% below EfficientMPT. The proposed methods therefore reach the state of the art and lead in the parameter budget.

Changes made

  • Section 4.1, Table 1: the three baseline columns above, with the best value per code and SNR in bold and the second best underlined
  • Section 4.2, Table 2: a new table of parameters and MACs with percentages relative to ECCT
  • Section 4.2, body text: two new paragraphs stating the conclusions of the parameter and MAC comparison
AuthorsOpenReview ↗

Response to Review 7yfg (2/3)

W3|BER only, BLER requested

Reviewer's comment

Results are limited to BER comparisons, which are not the main performance measure for error correction codes. Please provide BLER comparisons.

We accept this criticism. For error correction codes the block error rate is indeed more directly meaningful than the bit error rate, and this was an omission on our part.

We have added experiments on the block error rate. Table R3 gives the block error rate of the five decoders on one BCH code and one LDPC code. Both designs surpass the ECCT baseline; against the recent state-of-the-art models they reach a similar performance and exceed them in several cases. On LDPC(121,70), SECCT attains the lowest block error rate at all three SNRs while storing a fifth of the parameters of ECCT and CrossMPT.

MethodBCH(63,36) 4 dBBCH(63,36) 5 dBBCH(63,36) 6 dBLDPC(121,70) 3 dBLDPC(121,70) 4 dBLDPC(121,70) 5 dB
ECCT9.91e-021.98e-021.91e-032.97e-013.77e-021.23e-03
CrossMPT7.85e-021.43e-021.16e-031.63e-011.39e-022.31e-04
EfficientMPT1.01e-011.72e-021.77e-031.74e-011.61e-023.09e-04
SECCT (S1)8.28e-021.68e-021.25e-031.53e-011.24e-021.98e-04
HECCT (S2)8.41e-021.65e-021.44e-031.63e-011.31e-022.15e-04

Changes made

  • Appendix, new section "Block Error Rate": the explanatory text and the block error rate table covering one BCH code and one LDPC code

W4|AWGN only, fading channels requested

Reviewer's comment

Results are reported only over AWGN channels. It's pretty standard to provide similar comparisons also for fading channels.

We have added results for the Rayleigh fading channel. Because ECC decoders deployed over wireless links also see fading, we repeat the comparison under a Rayleigh channel, following the convention of the EfficientMPT appendix and of our earlier work: the received signal is y = h \odot x + z, where h is i.i.d. Rayleigh with unit scale on every bit, and the receiver performs no equalisation, so the AWGN-trained weights are used directly without retraining.

Table R4 reports one BCH code and one LDPC code. Under fading all decoders degrade together, since a bit that falls into a deep fade carries almost no reliability information and no method recovers it. Both of our designs nevertheless hold a stable performance on par with the baselines, and are better than them at some codes and SNRs.

MethodBCH(63,36) 4 dBBCH(63,36) 5 dBBCH(63,36) 6 dBLDPC(121,70) 4 dBLDPC(121,70) 5 dBLDPC(121,70) 6 dB
ECCT (N=6)3.5924.1514.9013.9364.9416.393
CrossMPT3.6884.3284.9874.3655.3077.300
EfficientMPT3.5954.2524.8194.3135.2787.213
SECCT (S1)3.6884.2584.9884.3105.4937.377
HECCT (S2)3.6714.2135.0674.2985.4117.279

Changes made

  • Appendix, new section "Rayleigh Fading Channel": the channel convention, the discussion of the results, and the Rayleigh table covering one BCH code and one LDPC code
AuthorsOpenReview ↗

Response to Review 7yfg (3/3)

Q1|How do the results of Tables 2, 3 and 4 extend to other codes?

Reviewer's comment

How would the results presented in Tables 2, 3 and 4 extend to other codes? Have you run any tests?

Two aspects are addressed below.

On the complexity conclusions. These conclusions are stable across codes; the detailed comparison is given in the complexity table. It is worth noting that we also ran experiments on longer codes. On LDPC(648,540), SECCT retains its parameter advantage, storing 0.79M parameters against 1.09M for EfficientMPT. At LDPC(1056,880) the order reverses: EfficientMPT attains 1.09M parameters and 1.65G MACs, compared with 1.66M and 7.58G for SECCT and 2.16M and 5.52G for HECCT. The reason is that the parameter count of EfficientMPT is independent of the codeword length and its attention scales linearly, whereas in the ECCT family the position embedding grows with the square of the sequence length and the attention is quadratic. Our methods therefore hold a clear advantage at short and medium blocklengths, while on long codes EfficientMPT retains a marked advantage. We report this as it stands.

On the ablation studies. We are in the process of adding the corresponding experiments and, time permitting, will report the results tomorrow, if possible.

Changes made

  • Section 4.2: a new paragraph comparing medium and long codes, giving the parameters and MACs for LDPC(648,540) and LDPC(1056,880)
  • The supplementary ablation results will be updated and submitted tomorrow

Q2|Why is the computational cost of SECCT twice that of ECCT?

Reviewer's comment

Can you please explain why the computational requirement of SECCT is double that of ECCT?

SECCT applies one and the same layer twelve times, and all twelve forward passes execute, so its MACs are exactly those of a twelve-layer unshared ECCT of the same depth. The factor of 2.0 in the paper is relative to the six-layer ECCT baseline, and is essentially the difference between twelve layers and six.

For certain memory-constrained edge scenarios, SECCT can keep both the stored parameters and the inference memory very low, and raise accuracy through iterated computation.

Changes made

  • Section 4.2, text after Table 3: a sentence added after the existing statement of the 2.0× MAC count, noting that the factor follows from the number of iterations (twelve against six layers) and that every layer has the same structure, so the computation grows with the number of layers while the stored parameters do not

Q3|Where does the reduction in computational cost of HECCT come from?

Reviewer's comment

Similarly, where is the reduction in the computational requirement of HEECT coming from?

Taking Polar(64,32) as an example, the reduction comes from two cuts inside each layer: grouped-query attention with G{=}2 narrows the key and value projections to a quarter of their full width, lowering the QKV projection of one layer from 4.72M to 2.36M MACs; the Bottleneck FFN reduces the intermediate width from 512 to 64, lowering the two FFN matrices from 12.58M to 1.57M, so that one layer falls from 21.23M to 7.86M. Even allowing for the fact that HECCT has twice as many layers as ECCT, its total computational cost is still reduced to 74% of the six-layer ECCT (94.4M against 127.4M).

Changes made

  • Section 4.2, text after Table 3: a paragraph added after the reduction figures for HECCT (parameters −40.5%, MACs −25.9%), explaining that each layer contains two reductions: grouped-query attention with G{=}2 narrows the key and value projections to a quarter of their width, and the Bottleneck FFN halves the intermediate width; together they lower the per-layer cost of the decoders in the table by more than half, and the twelve layers together still consume fewer MACs than the six-layer baseline

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.