- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The submission evaluates whether transformer-based models trained on Sudoku puzzles with one alphabet can generalize to Sudoku puzzles with a different alphabet. It finds that models fail to generalize to alphabets whose corresponding tokens have never been encountered during training. When the model is trained on diverse alphabets that mix a set of known symbols, they can they generalize to novel alphabet combinations.
Critical strengths
None
Critical weaknesses
- The main finding that models do not generalize to tokens not encountered during training is not surprising. There is a rich body of literature on compositional generalization and the role of symbol substitutivity that has long established similar findings but is hardly mentioned
- The paper has severe presentation issues, it uses very jargon-heavy language, making it difficult to follow.
Questions for authors
The classifier head’s disjoint per-alphabet output rows mean it cannot use a row it never trained; that much is architectural. What is not settled is how total the resulting failure is, since a shared encoder processing both alphabets could plausibly leak some above-chance signal even through untrained output rows.
Can you explain what reason there is to expect anything other than chance performance? What is meant with "a shared encoder processing both alphabets"? Has the encoder been trained on the held-out alphabet while the head hasn't?