- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses evaluation leakage and parametric-memory confounding in domain-reasoning systems by introducing a curriculum-driven construction framework and a provenance-auditable evaluation methodology. The authors formalize a domain curriculum via three inputs: (1) a ~200-word typed syllabus (premise) structured along four philosophical axes (ontology, epistemology, axiology, praxeology), (2) a textbook (domain corpus), and (3) problem sets (worked examples that specify in-scope question shapes). From these, the system builds a typed, provenance-carrying fact store (extracted via recipe-driven 3B-LLM reading in metabolic biochemistry and purely mechanical cue/structure mining in FTC consumer-fraud law) separated into a domain-agnostic mechanism kernel and a corpus-attested reading adapter. At answer time, a deterministic engine (, making zero LLM calls) assembles answers carrying verbatim receipts, logged assumptions (), or typed refusals (GAP), which are scored by a deterministic certificate checking both keyword match and premise-axis dependency grounding.
Critical strengths
In a conference culture where failed generalization tests and evaluation bugs are routinely swept under the rug, this paper sets a gold standard for honesty. The authors report discovering and fixing their own 13-point development leakage, label all iterative refinements as "fixed adaptive instruments" rather than true holdouts, report a failed mid-campaign iteration (), and feature a catastrophic out-of-sample failure in the Abstract and main text alongside a crystal-clear record-level diagnosis (Appendix H).
Critical weaknesses
- The paper is very poorly organized and hard to read, without any figures or graphs, and the tables are extremely out of shape. There is no formal mathematical or algorithmic specification of the kernel, the adapter schema, or the mutual-induction recipe extractor in §2–§3.
- The system is a classical symbolic knowledge base / regex-cue information extractor paired with a deterministic TF-IDF (rank-then-cap) template renderer, manually refined by a human+frontier-LLM team. The collapse on exam-2 in metabolism demonstrates the classic knowledge-acquisition bottleneck: even when the exact answer sentence is at IDF rank 1 in cases, a rigid record schema (reactions, regulatory, energetics) blinds the system to basic questions ("What organ stores glycogen?", "What hormone stimulates lipogenesis?") unless a human pre-defines a slot or inverse index for every relation type.
Questions for authors
- In your revision, can you move the core empirical tables (currently in Appendices B, C, D, and F) into the main text and provide a clear architectural diagram and formal specification of the kernel/adapter execution pipeline in §2–§3?
- Your RAG baseline uses a small 3B model (Qwen2.5-3B-Instruct) without an explicit abstention prompt, evaluated by an exact-keyword scorer that penalized 6 valid paraphrases in fraud law ( keyword vs. substantively defensible). Since your builder loop uses a frontier LLM to design repairs, how does a zero-shot frontier LLM (or Qwen2.5-3B with an explicit "cite verbatim receipts or output TYPED_GAP if absent from context" prompt) perform on the frozen exams and trap probes under the semantic rubric?