- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that while small LLMs could internalize basic coding skills and grammars via self-training, while it is not trivial for them to learn problem-solving skills for unseen situations, and would requires external constraints for decoding to handle complex structural errors. In experiments it is shown that there is a trade-off, where teaching the model with correct solving logic degrades the ability to write valid code.
Critical strengths
- The paper explains how the models internalize decoding constraints into their weights, instead of treating them as a black box. This may provide insights to readers.
- The tradeoff shown in the experiments may be interesting, possibly provide positive impact to the community.
Critical weaknesses
- The presentation and writing quality is, unfortunately, suboptimal and the paper requires a major revision to have a better clarity. For example, abstract provides only overly high-level concepts with terms never introduced (e.g., MLIR), them move to specific experimental setups. The paper, overall, should present what research question is asked, how is it solved, and what is insight. In the current status, it is really hard for readers to extract those information.
- Small language models would be able to learn basic coding skills through practice, but they would still need external constraints for decoding to avoid structural errors and scale to unseen tasks.
Questions for authors
- The trade-off between learning problem-solving logic and learning correct grammars is shown. Is this simply a limit of small language models, or merely a gap in the current training paradigm?