- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a Computational Reasoning Machine (CRM), which augments a transformer with a structured language called CRM ASSEMBLY. The model emits primitives that define variables and scopes, reusable functions, and deferred computations. These primitives determine which earlier reasoning tokens remain visible to subsequent tokens through dynamically constructed attention masks.
Critical strengths
Novel approach to structuring model reasoning through CRM ASSEMBLY, which supports scoped variables, reusable functions, deferred execution, and dynamic attention masking. The authors implement these mechanisms in a transformer and evaluate them across four reasoning benchmarks. Results show substantial reductions in attended tokens and shorter outputs, which means potential benefits in computational efficiency.
Critical weaknesses
Practical efficiency benefits are not sufficiently demonstrated. The paper reports reduced attention computation and theoretical parallelism, but does not establish corresponding end-to-end latency or throughput improvements. Accuracy also remains 1.2–6.7 percentage points below the CoT baseline after preference optimization. Evaluation lacks sufficient ablation studies to isolate the contributions of individual components, and syntax or runtime failures can truncate reasoning traces. The evaluation focuses on a single 4B model and a 4096-token context, so scalability and generalizability is unknown.
Questions for authors
Can you provide end-to-end inference latency, throughput, and memory measurements demonstrating that CRM delivers practical efficiency gains over CoT baselines, beyond reductions in attended tokens and theoretical parallelism?
What causes the remaining accuracy gap between CRM and CoT? How do you validate that generated computation graphs preserve correct reasoning dependencies despite syntax errors, runtime failures, and truncated traces?
We thank the reviewer for the thoughtful feedback.
See Inference FLOPs, Wall-Clock Speedup Results, and Peak Resident KV Cache Tokens and Memory in the overview comment.
See Significance: Generality in the overview comment.
Similar to LLM code generation, syntax and runtime failures can be addressed with additional training and with constrained decoding using methods such as SynCode[1].
See Feature Ablation Results in the overview comment.
We have updated the paper with results using 8192-token context length, which maintains the task-performance trends previously observed at 4096. Our work is the first to present a system that performs general computational reasoning in one model, and our evaluation demonstrates empirically that CRM reduces mean attended tokens and enables parallelism. These savings come from the semantics of CRM Assembly implemented through attention masks, which is present in all transformer models.
Figures 15 and 23 in the revised draft show the breakdown of the failure reasons for CRM and CRM-RL: 10.4-31.0% of CRM runs and 0.2-4.4% of CRM-RL runs failed due to incorrect syntax. Similar to LLM code generation, syntax errors can be addressed with additional training and constrained decoding. 16-29% of CRM runs and 13-32% of CRM-RL runs failed due to reasoning, which can be addressed with better quality training data annotations and additional training.
The CRM runtime aborts generation at syntax and runtime errors. Aborted and truncated traces are scored as-is using the official scorers of the benchmarks to extract an answer to score. Our evaluation methodology does not validate the traces for any other properties.
[1] Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. Syncode: LLM generation with grammar augmentation. arXiv preprint arXiv:2403.01632, 2024.