- The problem motivation and the necessity of the learned neural component are not sufficiently established.
My main concern is whether the proposed framework addresses a sufficiently meaningful machine learning problem in the evaluated settings. Both benchmarks involve explicitly defined mathematical operators, and CANOPY already assumes access to the expression tree and the exact operator semantics through its parser and closed-form composer. Under these assumptions, a purely symbolic evaluator can directly compute the correct answer without learning the atomic operations.
This raises a fundamental question: why is a neural network needed to approximate deterministic operations such as addition, multiplication, MIN, or MAX when their exact computation rules are already available? In fact, Section 5.2 reports that removing the learned residual entirely achieves perfect accuracy in the evaluated multiplication setting.
I agree that combining learned primitives with symbolic composition can be valuable, particularly when the primitives are unknown or must be inferred from noisy observations. However, the current experiments do not demonstrate such a setting. Consequently, the strong benchmark results do not clearly establish what additional capability the learned reducer provides beyond conventional symbolic computation.
- The experimental comparisons do not adequately isolate the contribution of the proposed method.
CANOPY is compared against end-to-end sequence models that must learn the compositional structure and operator behavior from raw tokens, whereas CANOPY receives an externally parsed tree and uses manually specified composition rules. These systems therefore solve substantially different problems, making the large performance and parameter-efficiency improvements difficult to interpret as evidence of a stronger neural architecture.
There are also concerns about the DeepMind Mathematics evaluation protocol. Section 5.2 uses heavily filtered integer-operand populations, with an acceptance rate of only 2.8% for the multiplication module. Furthermore, Table 2 evaluates examples generated by modules.train(ef), whereas the reference results in Saxton et al. (2019), Figure 4, are reported on the benchmark's interpolation test distribution. The official dataset implementation uses different entropy configurations for these two settings. Therefore, even the claimed population-matched comparison on add_sub_multiple does not appear to use an identical evaluation distribution.
Overall, the experiments demonstrate that the proposed hybrid system performs well on restricted deterministic reasoning tasks, but they do not sufficiently establish its advantage over the most relevant symbolic alternatives or support direct comparisons with end-to-end neural baselines.
Response to Reviewer rbxm
W1: repeated emphasis on headline numbers such as "90x".
We agree. "90x" is removed from the abstract. The abstract now states that the end-to-end baselines must also learn the tree.
W2: undefined notation.
We agree. modules.<module>.train(ef) and abs(pred - float(true value)) are removed. Section 5.2 and the Table 2 caption now say "training-generator questions at entropy setting (3, 10)", and the caption defines EM_0 as an exact value match.
Q1: table captions are too long.
We agree. The table captions are shortened. The details are in the text, and the LLM baseline protocol is in Appendix I.
W3 / Q2: reproducibility; build_atomic_reducer_dataset_v10.py.
We agree that a script name is not a description, and the script was missing from the anonymous release. Section 5.2 now describes the data instead. It is a directly sampled integer distribution of (op, a, b) tuples, seed 0, not drawn from the DM Math generator. The script is now in the anonymous repository, with a README entry.