- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies how augmentation repairs compositional generalization failures and how compression affects whether repair occurs in the encoder or decoder.
Critical strengths
The experiments go beyond showing that augmentation helps. Sections 4.2–4.5 vary coverage, examples per combination, compression, and encoder freezing to examine when and where repair occurs. The robustness checks are informative. The appendix tests input noise, an MLP encoder, and a Transformer decoder, and reports where the original findings fail to reproduce. The frozen-encoder results have practical value. Decoder-only adaptation recovers about 79% of the repair gain on the continuous-feature bridge and 88.5% with the end-to-end CNN.
Critical weaknesses
-
Proposition 3 misuses Cover’s theorem. Section 4.6 turns the fraction of separable binary labelings into the probability that one particular code is separable from the other eleven. These are different quantities.
-
The necessity claim is stronger than the freezing results support. At , frozen and trainable encoders reach 0.169 and 0.691 accuracy, respectively. This is a substantial gap, but the actual decoder is a nonlinear LSTM. Linear inseparability in the input code space does not establish that decoder-only repair is impossible; the gap could also reflect optimization or capacity limits.
-
The headline transition is sensitive to the setup. Appendix C reports that adding noise with removes the crossover, while the Transformer decoder never shows freezing outperforming full training. The continuous-feature compression scan also produces a roughly constant freezing deficit. These results narrow the mechanism considerably beyond what the title and “full mechanism” language suggest.
Questions for authors
-
Can Proposition 3 be corrected to address one-versus-rest separation, with explicit assumptions on the code distribution?
-
Does the freezing gap persist with longer training, tuned optimization, and a stronger decoder?