- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper adapts recursive training and inference techniques, including deep supervision, truncated gradients, adaptive halting, and discrete deduction lattices, to message-passing networks. Experiments are mostly focused on Sudoku, while it has also some additional on mazes, N-Queens, and SAT. A central finding is that sensitivity to gradient truncation depends on the graph’s propagation distances.
Critical strengths
The paper demonstrates strong parameter efficiency and provides useful comparisons between graph structures, training procedures, and inference strategies. The analysis of gradient truncation offers insight into how graph connectivity affects learning. Experiments on multiple tasks, including transfer to larger N-Queens and SAT instances, strengthen the contribution beyond Sudoku alone.
Critical weaknesses
The abstract could state the central goal more directly and clarify how message passing supports algorithmic reasoning; the broad title could better reflect the controlled puzzle setting. Although the related work is substantial, it omits "Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error", "Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles", and "Learning to Solve Constraint Satisfaction Problems with Recurrent Transformer". Sudoku training uses only 1,000 base puzzles with augmentation, leaving broader generalization uncertain. The model-selection protocol also needs clarification, and minor presentation issues include an unresolved “Sec. ??” (line 296) reference and the semicolon before “None” at line 193.
Questions for authors
-
Could you evaluate online training with freshly generated Sudoku puzzles, for example using SudokuPy, as in Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error?
-
Given the use of only 1,000 base training puzzles, what evidence rules out overfitting to this dataset despite augmentation? If computationally feasible, evaluation on a broader Sudoku distribution and independent datasets with different difficulty levels would strengthen the generalization claims.
-
Could you also clarify the discrepancies in model selection and validation-set sizes, and compare methods under matched inference budgets?
-
Section 5.1 states that model configurations were selected using test accuracy, whereas Appendix S3 describes checkpoint selection using validation accuracy. Was the test set used for any configuration or model selection? Also, could you clarify why Section 5.1 reports 300 validation puzzles while Appendix S3 reports 500?
-
Finally, how do comparisons change when inference budgets, including the 64 stochastic chains, are matched?