- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work targets the parameter cost of the Error Correction Code Transformer (ECCT) and its foundation-model successor (FECCT). The authors pursue two directions: reusing Transformer layers across depth, and Grouped-Query Attention (GQA) with a bottleneck FFN, corresponding to SECCT and HECCT respectively.
In the reported experiments, SECCT achieves better performance than ECCT while storing only about 18% of its parameters, and HECCT uses fewer parameters than ECCT while implementing a deeper model (12 layers vs. 6). SECCT and HECCT deliver comparable results.
Critical strengths
This work answers a question: how an ECCT-style model can be simplified, both along depth (layer reuse) and within each layer (simplified attention and FFN architecture).
Critical weaknesses
- Missing comparisons with recent SOTA methods. While ECCT can be regarded as an early work on parity-check-matrix-masked Transformers, several subsequent methods, including CrossMPT, EfficientMPT, and SBECCT, are neither sufficiently discussed nor included as baselines. This makes the related-work section incomplete and, more importantly, makes it difficult to assess whether the proposed methods actually improve upon the current state of the art. Based on the reported results, it is also unclear that the proposed approaches consistently outperform these more recent models.
- The methodological novelty appears limited. In particular, given the existence of EfficientMPT and other closely related architectures, the proposed contribution seems largely incremental. The paper should provide a much clearer comparison with these prior methods and explicitly identify which architectural or algorithmic components are genuinely novel, rather than variations of existing designs.
- The connection between SECCT and HECCT is weak. The two proposed architectures appear to be motivated and designed largely independently, and the manuscript does not provide a sufficiently strong conceptual or technical connection between them. In the current presentation, combining them into a single paper feels somewhat artificial; each method could arguably constitute an independent work. A clearer unifying perspective is needed to justify presenting both methods together.
- The paper lacks an experiment that directly connects SECCT and HECCT. A natural experiment would be to construct a hybrid SECCT+HECCT model, for example by reusing or sharing HECCT layers/components within SECCT. Such an experiment would help determine whether the two proposed approaches are complementary and would provide stronger empirical support for treating them as parts of a unified framework.
Questions for authors
Minor issues:
- Please use
\citetand\citepappropriately according to their grammatical roles in the sentence. The manuscript currently appears to use\citetfor nearly all citations. This makes the prose awkward and reduces the overall readability and polish of the paper.
7 comments
Response to Review vZ7X (2/3, W2)
W2|Limited methodological novelty, incremental with respect to EfficientMPT
Reviewer's comment
The methodological novelty appears limited. In particular, given the existence of EfficientMPT and other closely related architectures, the proposed contribution seems largely incremental. The paper should provide a much clearer comparison with these prior methods and explicitly identify which architectural or algorithmic components are genuinely novel, rather than variations of existing designs.
Regarding the difference from these methods, we consider the contributions to be unlike one another: each of them modifies a different component, and they therefore cannot substitute for each other. For ease of comparison, the table below lists the core contribution of each method and the component it modifies; the same comparison is given in Section E of the appendix.
| Method | Core contribution | Component modified |
|---|---|---|
| CrossMPT | Updates the magnitude and syndrome embeddings separately through masked cross-attention, so that the two kinds of information are iterated in their own channels | attention form (self → cross) |
| EfficientMPT | EEC attention: keeps only the query–key interaction, replaces the per-position query with a global query vector, and removes the value projection | how attention is computed in each layer |
| AECCT | Ternary weight quantization (AAP), head-partitioned self-attention (HPSA), spectral positional encoding (SPE) | weight precision and attention structure |
| SECCT | Replaces the independently stored parameters of each layer with a single shared parameter set applied repeatedly along the depth: every layer performs the same mapping, so the number of iterations is decoupled from parameter storage, and an arbitrarily deep decoding network can be emulated at constant storage | parameter storage between layers |
| HECCT | Compresses two sub-structures within a layer at the same time: on the attention side it narrows the key and value projections and shares one set of key–value projections across several query heads; on the feed-forward side it compresses the hidden width, so that the intra-layer nonlinearity operates in a lower-dimensional subspace | key–value projection width and feed-forward hidden width within a layer |
It can be seen that CrossMPT, EfficientMPT and AECCT all act on how computation is performed within a layer; their number of layers and embedding dimension are unchanged, so their storage still grows linearly with depth. In particular, the parameter count of SECCT does not grow with depth, which differs markedly from the optimisation direction of the prior papers: it does not lower the cost of each layer but removes the per-layer storage itself.
In summary, the difference between this paper and prior algorithms lies in two respects: the optimisation direction and the optimisation object.
For SECCT the direction differs from prior work: prior methods optimise the resource usage within a layer, whereas SECCT addresses the resource usage between layers, so its parameter count does not grow with depth.
For HECCT the direction is the same as in prior work, all acting on the resources within a layer, but the object differs: prior methods compress the way attention is performed, the weight precision or the feed-forward structure, whereas HECCT compresses the projection width between keys and queries and the hidden width of the feed-forward.
Changes made
- Section 2, Related Work: a new paragraph introducing the mechanisms of CrossMPT, EfficientMPT and AECCT
- Section 4.2, Table 2: a new table of parameters and MACs, allowing the storage and computation of the three compression schemes to be compared item by item
- Section 3, SECCT and HECCT subsections: state that the two forms of compression act between and within layers, respectively
- Section E of the appendix: a new section containing a comparison of the component each method modifies and the resource it reduces
Response to Review vZ7X (3/3, W3&W4)
W3|The connection between SECCT and HECCT is weak
Reviewer's comment
The connection between SECCT and HECCT is weak. The two proposed architectures appear to be motivated and designed largely independently, and the manuscript does not provide a sufficiently strong conceptual or technical connection between them. In the current presentation, combining them into a single paper feels somewhat artificial; each method could arguably constitute an independent work. A clearer unifying perspective is needed to justify presenting both methods together.
We accept this. The presentation in the original submission did not make the relation between the two clear. Section E of the appendix discusses this point specifically, and the content is summarised below.
SECCT and HECCT represent two different optimisation directions: one addresses the resources between layers, the other the resources within a layer.
SECCT addresses the resources between layers. It replaces the independently stored parameters of each layer with a single shared parameter set applied repeatedly along the depth; every layer performs the same mapping, so parameter storage is decoupled from the number of iterations, and an arbitrarily deep decoding network can be emulated at constant storage. What it reduces is the parameter storage between layers.
HECCT addresses the resources within a layer. It compresses two sub-structures within a layer at the same time: on the attention side it narrows the key and value projections and shares one set of key–value projections across several query heads; on the feed-forward side it compresses the hidden width, so that the intra-layer nonlinearity operates in a lower-dimensional subspace. What it reduces is the resource occupied by each sub-structure within a layer.
These two optimisation directions are orthogonal, and we present them separately in order to provide a solution for each direction:
- extreme parameter compression with a constant parameter count, at the cost of higher computation;
- a simultaneous reduction of both parameters and computation.
This better demonstrates the effect of the two optimisation directions, and is of practical value for different deployment scenarios.
Changes made
- Section E of the appendix: a new section explaining that the two architectures act on different resources and are orthogonal, together with a comparison against the prior methods
- Section 3: the SECCT and HECCT subsections state that the compression acts between and within layers, respectively
- Section 4.2, Table 2: the storage and computation effects of the two forms of compression are placed side by side in one table
W4|Missing experiment directly connecting the two
Reviewer's comment
The paper lacks an experiment that directly connects SECCT and HECCT. A natural experiment would be to construct a hybrid SECCT+HECCT model, for example by reusing or sharing HECCT layers/components within SECCT. Such an experiment would help determine whether the two proposed approaches are complementary and would provide stronger empirical support for treating them as parts of a unified framework.
Our intention was to present the two schemes separately, as solutions for different application scenarios. For example, the constant-level parameter storage of SECCT suits edge devices with severely limited storage and memory, where performance is traded for by increasing the number of iterations; if the storage budget is somewhat more generous, HECCT can be used to further reduce the computational cost and improve performance.
We accept the reviewer's point that an experiment connecting the two is necessary, since otherwise it causes confusion for readers. We have made every effort to run the simulations related to this response, and will revise the paper and add a comment at the first moment the results are available.
Response to Review vZ7X (1/3, W1&Minor)
W1|Missing comparison with the recent state of the art
Reviewer's comment
We accept this criticism. The failure to include these three works in the original submission was an oversight rather than a deliberate choice.
The gap is now closed: a new paragraph in Section 2 (Related Work) discusses CrossMPT, EfficientMPT and AECCT; Table 1 in Section 4.1 gains three baseline columns covering all 13 codes at 4/5/6 dB; and a new Table 2 gives the parameters and MACs of these methods.
For space, the table below gives only representative codes and an ellipsis marks the rest:
and the corresponding parameters and MACs:
The results indicate that the proposed schemes reach a performance comparable to the current state of the art, with savings in both parameters and computation. Both of our designs surpass the earlier decoders, namely the BP-based methods together with ECCT and FECCT, on all 39 code–SNR settings; against the recent AECCT, CrossMPT and EfficientMPT, they reach a similar performance and exceed them in a number of cases. In parameters, SECCT stores the fewest on every code, 82–83% below ECCT and 80–81% below EfficientMPT, at the cost of a MAC count twice that of ECCT; HECCT stores 40–41% fewer parameters than ECCT and 34–35% fewer than EfficientMPT, and its MACs are the lowest of all compared methods on every code. The proposed methods therefore reach the state of the art and lead in the parameter budget.
Changes made
Minor|Inappropriate use of
\citetand\citepReviewer's comment
We have distinguished the two citation forms according to their grammatical roles:
\citetwhere the author name forms part of the sentence, and\citepfor parenthetical citations. All citations in the body text have been adjusted accordingly.Changes made
\citetwhere the author name is part of the sentence,\citepfor parenthetical citations