acceptodds
Under review as a conference paper at ICLR 2027

Computational Understanding: Curriculum-Driven Construction and Provenance-Auditable Evaluation of Domain Reasoning

Abstract

Evaluations of LLM-based domain-knowledge systems are vulnerable to a quiet failure: the score can measure the substrate's parametric memory, or the evaluators' own leakage, rather than the constructed artifact. Auditing our own harness with a grounded rubric exposed two such channels — a hand-authored fact table and a construction cache — that supplied 13 of 50 development answers; we removed both, added specified controls, and rebuilt evaluation around a curriculum specification — a typed premise (syllabus), a corpus (textbook), and worked examples (problem sets) — scored by a deterministic certificate with the adaptivity of every later step disclosed under an explicit exposure taxonomy. On first evaluation of frozen, receipt-ruled, human-authored exams, the constructed stores lose to premise-conditioned retrieval-augmented prompting in both domains — 7/27 vs 17/27 (biochemistry) and 14/24 vs 15/24 (consumer-fraud law) — but every miss carries a provenance trail from which the builder diagnosed an addressable defect class. Across pre-registered repair iterations on those same fixed instruments (adaptive-reuse exposure disclosed throughout), the stores reach 20/27 and 23/24 with zero answer-time model calls and, under the stated confabulation definition, no invented text or sources identified in the audited campaign-final and fresh-exam outputs. We then ran a pre-registered zero-adaptivity generalization test: one fresh exam per domain, single scored runs. The fraud system met its pre-registered floor (10/13); the metabolism system failed decisively (2/19), and post-hoc diagnosis identifies a representation limitation — the fresh exam demands relation classes the practice-shaped record ontology never carried, while the engine's failure mode stayed typed refusals and grounded near-misses, with all designed probes refused. The curriculum specification, the provenance-auditable answer artifact, the disclosed repair loop, and the generalization test that binds its own authors are the contributions; the domain systems are the worked examples.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
understanding, curriculum-driven, provenance-grade

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_aEsMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses evaluation leakage and parametric-memory confounding in domain-reasoning systems by introducing a curriculum-driven construction framework and a provenance-auditable evaluation methodology. The authors formalize a domain curriculum via three inputs: (1) a ~200-word typed syllabus (premise) structured along four philosophical axes (ontology, epistemology, axiology, praxeology), (2) a textbook (domain corpus), and (3) problem sets (worked examples that specify in-scope question shapes). From these, the system builds a typed, provenance-carrying fact store (extracted via recipe-driven 3B-LLM reading in metabolic biochemistry and purely mechanical cue/structure mining in FTC consumer-fraud law) separated into a domain-agnostic mechanism kernel and a corpus-attested reading adapter. At answer time, a deterministic engine (U0U_0, making zero LLM calls) assembles answers carrying verbatim receipts, logged assumptions (UAU_A), or typed refusals (GAP), which are scored by a deterministic certificate checking both keyword match and premise-axis dependency grounding.

Critical strengths

In a conference culture where failed generalization tests and evaluation bugs are routinely swept under the rug, this paper sets a gold standard for honesty. The authors report discovering and fixing their own 13-point development leakage, label all iterative refinements as "fixed adaptive instruments" rather than true holdouts, report a failed mid-campaign iteration (7→67 \to 6), and feature a catastrophic 2/192/19 out-of-sample failure in the Abstract and main text alongside a crystal-clear record-level diagnosis (Appendix H).

Critical weaknesses
  1. The paper is very poorly organized and hard to read, without any figures or graphs, and the tables are extremely out of shape. There is no formal mathematical or algorithmic specification of the kernel, the adapter schema, or the mutual-induction recipe extractor in §2–§3.
  2. The system is a classical symbolic knowledge base / regex-cue information extractor paired with a deterministic TF-IDF (rank-then-cap) template renderer, manually refined by a human+frontier-LLM team. The 2/192/19 collapse on exam-2 in metabolism demonstrates the classic knowledge-acquisition bottleneck: even when the exact answer sentence is at IDF rank 1 in 15/1715/17 cases, a rigid record schema (reactions, regulatory, energetics) blinds the system to basic questions ("What organ stores glycogen?", "What hormone stimulates lipogenesis?") unless a human pre-defines a slot or inverse index for every relation type.
Questions for authors
  1. In your revision, can you move the core empirical tables (currently in Appendices B, C, D, and F) into the main text and provide a clear architectural diagram and formal specification of the kernel/adapter execution pipeline in §2–§3?
  2. Your RAG baseline uses a small 3B model (Qwen2.5-3B-Instruct) without an explicit abstention prompt, evaluated by an exact-keyword scorer that penalized 6 valid paraphrases in fraud law (15/2415/24 keyword vs. 21/2421/24 substantively defensible). Since your builder loop uses a frontier LLM to design repairs, how does a zero-shot frontier LLM (or Qwen2.5-3B with an explicit "cite verbatim receipts or output TYPED_GAP if absent from context" prompt) perform on the frozen exams and trap probes under the semantic rubric?
Reviewer_2c7uOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The author proposes a Curriculum-Driven Construction. They work on questions which has three inputs: premise, corpus and examples. After several rounds of “pre-registered” fixes, it climbs to 20/27 and 23/24.

Critical strengths

Reading this article, I found one design element in their metho could be valuable: the principle of "admitting when something is unknown." Instead of fabricating an answer, the system either provides a response based on stored records, complete with citation, or explicitly states that it lacks the necessary information. But, this is not a new idea.

Critical weaknesses

I will begin by stating my overall impression, followed by the specific reasons for it. This is a piece of coursework-level writing. This is an "auto-research" work, that bears clear, extensive hallmarks of having been generated by tools like Codex or Claude. I am not against using AI to assist with scientific research. In fact, I am also a user of it myself. However, relying on it extensively without proper review/revise can introduce many of the AI’s inherent flaws into one’s writing/design, such as an overabundance of obscure terminology, excessive use of em-dashes and bold text, a lack of logical flow, and abrupt, jarring transitions.

The following are specific and critical weakness:

  1. The format and wiritng is not readable. - First, this is not ICLR format. There are some horizontal lines appearing between the paragraphs. - As for the writing, the logic is not good. For example, in the introduction, the authors said 20/20 while the header collapses to 0/20. I have no idea about what does this fraction mean and what is 20? The same issue appears many more times later on. - No figures. This is not a theoretical paper, but rather one focused on methodological innovation. The absence of figures makes it difficult to grasp the method quickly.

  2. The scale. The scale is too small to support any conclusion. The coverage of domain (only 4), the questions each domain have (only 27, 24, 19, 13), the experiments runs only once, only Qwen2.5-3B (this model is too old, Qwen 3.5, Qwen 3.8 are avaiable!).

  3. The method does not have a good result. The headline 7->20 and 14->23 gains come from repeated fixes made after seeing per-item results on the same frozen exams, which the authors themselves call a “fixed adaptive instrument.” That is test-set tuning, so beating an untuned RAG baseline this way shows nothing, yet the abstract and contributions still say good with it.

Questions for authors

Authors should be more honest about their use of AI; the extent to which they use AI far exceeds what they claim in their paper.

Reviewer_Sk9NOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that evaluations of LLM-based domain-knowledge systems can measure parametric memory or harness leakage rather than the constructed artifact. The authors build a fact store from a short typed premise (syllabus), a corpus (textbook) and worked examples (problem sets), with provenance on every record and a deterministic keyword certificate for scoring. A self-audit finds two leakage channels that inflated their development score by a net 13 of 50 points. On frozen human-authored exams in metabolic biochemistry and consumer-fraud law, the store first loses to premise-conditioned RAG on the same 3B model (7/27 vs 17/27, 14/24 vs 15/24), reaches 20/27 and 23/24 after repair iterations on the same exams, and on fresh exams scores 10/13 (fraud) and 2/19 (metabolism).

Critical strengths

The grounded certificate and the self-audit make a concrete point about evaluation leakage. The same 3B model scores 20/20 under conclusion-only scoring with either the full premise or a bare header, but 20/20 versus 0/20 under the grounded rubric, and the leakage in Section 4.3 is accounted for exactly. This is a useful worked example for the RAG and knowledge-system evaluation community.

Critical weaknesses
  1. The evidence does not support claims beyond the specific instruments. Each exam has 13 to 27 items written by one author, every result is a single run, and the headline scores of 20/27 and 23/24 come from repairing the system against the same exams. The only held-out test reports no RAG or closed-book baseline, and the metabolism system drops to 2/19 there. Under the secondary audit rubric, the fraud gap over RAG also shrinks from eight points to two (23/24 vs 21/24).

  2. The contribution is a human-directed engineering process rather than a transferable ML method, and the paper is very hard to read. Every gain comes from a repair loop run by a human with a frontier LLM, and the ontology and recipes are rewritten per domain. The paper relies on private vocabulary ("one-way doors", "receipt-ruled", "OEAP axes") and arrow-chained score trajectories, and its fit with ICLR is weak.

Questions for authors

The following clarifications could change my assessment.

  1. Can you report the RAG and closed-book arms on both fresh exams under the same scorer?
  2. What is the metabolism comparison with RAG under a scorer that does not reward verbatim substrings?
  3. Which parts of the pipeline are automatic and domain-independent, and what would the system score without the human-plus-LLM builder?
  4. Do the conclusions hold with a larger model than Qwen2.5-3B-Instruct?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.