- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
KNWM is an explicit knowledge base plus a solver, not a neural network. Knowledge is stored as nested "knowledge neurons": records with an identity, type, content, slots, provenance and validation fields, held in five scientific layers (property, causal relation, mathematical pattern, composition, physical law) alongside layers for events, problems and planning. To learn a relation, KNWM selects properties that change with the target and, where it can control a simulator, perturbs them to isolate their effect; it then proposes a formula by symbolic search, by analogy to a stored structure, or by composing fragments, holds the structure fixed, refits the constants, and admits it by fitting error, parameter stability or held-out performance depending on the route. To solve a problem it decomposes backwards from the goal, activates relations forwards once their inputs are known, and uses the computed quantities as event costs.
Critical strengths
-
The problem is worth working on: keeping learned relations in a form that is persistent, inspectable and reusable, rather than re-derived for every task (L040–L045).
-
The supplementary artifact is complete and honest. It contains the scientific, memory, event-planning and perception code, the experiment generators and scorers, the knowledge records, a launcher with one command per experiment in the paper, a hash manifest, and a release note that states which results changed when defects were repaired. It also states what it does not cover: the artifact excludes the desktop application, and its own report says the PhysicsQA run reproduces an internal adjudication rather than independent grading.
Critical weaknesses
-
The per-layer semantics exist only in the code. The paper states the record fields (L202–L211) and gives per-type content and output contracts in Table A.1 (L649–L661), but it gives no per-layer contract: no input format, output format or computation for any of the five scientific layers, and no pseudocode. The update rules promised at L158–L159 and L704–L705 are supplied for the physical-law layer only (L160–L162), so the property, causal-relation, mathematical-pattern and composition layers never receive one in the paper. The definitions are recoverable from the released artifact, but a reader of the paper cannot tell how a stored entry changes as new evidence arrives, or what the representation does that a typed dependency graph would not.
-
The main text contains no table and no measured plot. All nine tables are in the appendix, namely A.1 (L649), D.1 (L959), D.2 (L973), D.3 (L1053), D.4 (L1081), D.5 (L1106), D.6 (L1119), D.7 (L1155) and E.1 (L1197), and every figure is a schematic. Headline numbers therefore appear in Section 5 only as prose: the four outcomes of the 192-task experiment are sentences at L396–L399, with the tabulated form, including the new-seed rows, confined to Table D.4 (L1084–L1091). The same pattern holds for the admission comparison (prose L355–L359, table D.1) and the planning comparison (prose L388–L394, table D.3). This is presentation rather than evidence, but it makes the paper slow to check and hides the pattern across conditions.
-
Repairs post-date the paper, and the paper does not say which implementation produced its numbers. The release notes list four families of defects repaired in the artifact: a knowledge-applicability condition that wrongly required an unrelated label, formula learning, where candidate selection, fragment interfaces and coefficient calibration were corrected, an adaptive-control update that did not require observability, and the visual frontend. The notes state that the version used for the paper's control experiment is retained separately, and name two outcomes that move under repair: completion from 78/100 to 79/100 in the control suite, and visual accuracy from 96/100 to 100/100 on the original seed and 99/100 on a new one. For the law-learning slice the artifact provides repaired-code commands whose expected outcomes match the paper, at 20 accepted with 19 passing against 30 accepted and 20 passing for training-only admission, so those numbers appear unaffected; but the paper itself reports neither the repairs nor which version produced which result, and a reader cannot tell which reported numbers came from code that has since been corrected. The same release concedes two further limits that the paper does not: the 900-task set cannot be regenerated from its seed, because retrieval generation consumes randomness before later categories, and the language-model baselines are re-scored from archived responses rather than re-queried, with their snapshots not fully pinned (L1240–L1241).
-
The paper does not position itself against the closest existing instance of contribution #1. The reference list contains no proof-assistant or interactive-theorem-proving work: no Lean, Coq, Isabelle or machine-checked library (L505–L555). Such a library stores reusable statements together with the premises under which they apply, and faces the same retrieval problem of selecting statements whose premises are satisfied in the current context, which makes it the nearest neighbour of a stored relation with explicit conditions (L053, L202–L204). The related work covers frames, STRIPS, SHOP2, SINDy, PySR and retrieval-augmented generation (L062–L086) but not that literature, nor symbolic-regression work on reusing previously discovered expressions. The contrast would have helped, because KNWM's relations are empirical and admitted over a measured range (training ≤3%, validation ≤5%, stability ≤0.20, L950–L953), a caveat the paper states for fitting (L706–L707) and for provenance (L1258–L1263), but never against formal derivation from stated premises.
-
The comparisons that should test the claim are ties. The symbolic comparator solves all 300 composition tasks that KNWM solves (L370–L372), passes all 6,000 prediction checkpoints (L376–L377), and scores 900/900 on the 900-task set (L1246–L1247). In the planning experiment, validated reuse reaches 17/24 optimal plans, the same as a conventional target index given identical knowledge (L392–L394, L1055–L1059); the gain over having no reuse, at 6/24, shows that the learned relations matter, not that the knowledge-neuron representation does. The authors say as much themselves at L392–L394. As written, the paper supports the claim that a pipeline which learns relations can drive planning, but not that this architecture is what makes it work.
Questions for authors
-
Can the paper state, for each of the five scientific layers, the input format, the output format, the computation, and the update rule, so that the specification is in the paper rather than only in the released code?
-
Which implementation produced each reported number, the version described in the paper or the repaired version in the artifact? The release notes show that the control and visual outcomes move under repair; can the paper state the equivalent for Section 5.1, where the artifact's map expects the law-learning numbers to be unchanged?
-
What would a comparison against a language model coupled to a proof assistant and its library, for example Lean with Mathlib and premise retrieval, show, and what does a knowledge-neuron record carry that a checked lemma with typed hypotheses does not?
-
Can the results be tabulated in the main text, with conditions as rows and the ablations from Appendix D summarised alongside them?
-
Given that a symbolic comparator with the same knowledge matches the reported results, what experiment would separate the contribution of the representation from the contribution of the relations stored in it?
Author Response to Reviewer FzTT
Thank you for the audit. I address each issue below.
Q1 Layer contracts
Knowledge tokens represent typed quantities (F, m, a) and recursive fragments (ma). They compose scientific, event and problem neurons; the five layers subdivide scientific neurons. The record is K=(id,name,type,category,hierarchy,slots,content,connections,provenance,validation,metadata).
Each layer specifies input -> output; computation; update:
Property: name/type/unit/range/role -> property/hierarchy; register/query; update supplied fields, append contexts and synchronize hierarchy.
Causal relation: cause/effect/context/evidence -> directed record; related-quantity lookup; match cause/effect/context, retain maximum recorded confidence and update the label.
Pattern: form/role/fit -> pattern/fragment; register fitted structures; match name/role, retain maximum fit and append contexts.
Composition: rule/conditions/tested count -> template; retrieve aggregation rules; retain maximum tested count, append contexts and OR the legacy flag.
Law: expression/target/parameters/conditions/evidence -> executable relation; retrieve/evaluate; require expression-specific evidence for empirical replacement.
Confidence and legacy flags are metadata, distinct from empirical validation. Execution needs known inputs, compatible object/state bindings and satisfied conditions; unsupported cycles create no values. Replacement requires expression/target-specific evidence and atomic update; revisions refresh indexes and derived quantities.
Q2 Result provenance
Submitted -> repaired results: control 78/100 -> 79/100; original-seed vision 96/100 -> 100/100; new-seed vision 99/100. Original 36-task results are unchanged: validated admission stores 20, with 19 passing the distant test; training-only admission stores 30, with 20 passing. It uses bounded expression search, separately from the repaired PySR wrapper.
Replaying the archived 900 inputs scores 898/900; inputs, scorer and both failures are retained. Replay differs from same-seed regeneration because generation consumes randomness before later categories. Historical LLM scores rescore archived responses, with incomplete snapshot pinning. Frozen packages and source/data fingerprints identify versions; later additions are recorded separately.
Q3 Proof assistants and typed libraries
A fixed local LLM retrieves premises from 12 relations before arithmetic execution and Lean 4.19/Mathlib checking. On 100 composition, applicability, binding, cycle and missing-premise cases, KNWM and a typed graph each score 100/100; LLM retrieval plus Lean scores 96/100. Correct outcomes include required non-completions; 26 numerical consequences have kernel-checked proofs. This tests a controlled library, not unrestricted theorem proving. KNWM tracks empirical validity/revision; formal checking validates consequences under hypotheses. Typed libraries can carry this metadata too. My contribution connects learned relations to execution, revision and planning.
Q4 Main-text presentation
Principal baseline comparisons, progressive-reuse results and procedure-connection ablations are now in the main text, with layer contracts beside the method. Text points to these tables; detailed configurations, per-case results and reproduction records remain in the supplement.
Q5 Isolating the connections
With identical relations, facts, conditions and execution, KNWM and a typed graph each score 180/180 lifecycle cases. Removing binding fails all 30 binding cases; removing cross-relation reuse fails all 30 composition cases; removing revision refresh fails both 30-case revision groups.
I then varied representation and learned guidance with formulas, planner, executor and limits fixed. Completion, transition attempts and time are:
KNWM with guidance: 40/40, 343, 8.932 s.
KNWM without: 24/40, 18,134, 40.709 s.
Typed graph with identical guidance: 40/40, 343, 6.648 s.
Typed graph without: 24/40, 18,134, 38.080 s.
Attempts/times cover 40 valid tasks plus eight safety/integrity cases. Completion uses the first of two repetitions; times average both. Four procedures were learned from 12 execution traces; answers/intermediate values were removed before reuse.
Within KNWM, connections raise completion from 60% to 100%, reduce attempts by 98.11% and total time by 78.06%; time falls 45.94% on the same 24 tasks completed by both. The graph control reproduces the gain, identifying learned executable connections as the effective mechanism linking scientific computation, event procedures and problem goals. Post-submission procedure-learning additions are versioned separately.