acceptodds
Withdrawn from ICLR 2027

Distilling Tacit Procedural Knowledge: From Scholarly Literature to Skill-Guided Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) possess extensive factual knowledge but often struggle with expert problems that require coordinated use of heterogeneous evidence. We argue that a key missing component is tacit procedural knowledge—the knowledge of selecting, integrating and revising evidence. We propose Explicit-to-Tacit-to-Skill (E2T2S), a framework that extracts structured knowledge from scholarly papers, aggregates it across each scholar's research to recover reusable procedural patterns, and consolidates them into an Expert Skill Library for dynamic skill-guided reasoning. We evaluate E2T2S on Chinese Chu bamboo-slip character interpretation, where models must infer a modern Chinese character from its glyph image, textual context, external evidence, and scholarly literature. On a controlled held-out evaluation set, our method outperform both generic chain-of-thought and retrieval-augmented baselines. Our results demonstrate that scholarly literature can serve not only as a source of knowledge, but also as a source of reusable expert procedures for LLM reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Tacit Knowledge, Knowledge-Augmented Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cfHROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes E2T2S, distilling reusable expert procedural knowledge from scholarly literature. Instead of treating literature merely as a source of knowledge, it aggregates patterns across papers to identify stable reasoning habits and decision-making strategies, constructs an Expert Skill Library from these patterns, and dynamically selects relevant skills during inference. The authors evaluate the framework on Chinese Chu bamboo-slip character interpretation and compare it with methods such as CoT and RAG.

Critical strengths

The idea is insightful. The paper argues for explicitly extracting how experts use knowledge—for example, how they select, combine, and compare evidence, and when they revise an existing judgment. Distilling such reasoning patterns, accumulated through long-term professional practice, into reusable skills is intuitive and worth exploring.

Critical weaknesses

The paper has an appealing motivation, but some claims appear overstated. All experiments are currently focused on Chu bamboo-slip character interpretation. This is a suitable testbed, indeed. However, if the authors intend E2T2S to be a general paradigm, it would be persuasive to discuss other domains where E2T2S may apply. Even within character interpretation, experts may rely heavily on intuition and inspiration. It would therefore be helpful to provide one concrete example of a Skill in the library, showing what it looks like in practice and how its use improves the performance.

The implementation details are insufficiently specified:

(1) E2T2S contains multiple stages, but it is unclear how these components are actually implemented. Are they realized purely through prompting, or do they involve additional tools, models, or algorithms?

(2) The CoT and RAG settings are also under-specified. Nowadays CoT is often necessary part of a more elaborate system or pipeline, so the complete prompts should at least be provided. For RAG, details such as the retriever/embedding model and chunking strategy are missing, and without metrics such as Recall are reported. Given that Qwen3-VL-8B-Instruct is not a frontier model and may depend more heavily on a well-tuned RAG pipeline, the negative gain of RAG in Table 1 makes these implementation details particularly important to clarify.

Some of the experimental results appear somewhat anomalous:

(1) In Table 2, CoT even leads to performance degradation on categories such as Supportive Evidence. Under Single Evidence Type, RAG+CoT performs worse than RAG, which is inconsistent with the trend of CoT outperforming Base in the same category.

(2) In Table 3, the accuracy of w/o Tacit is even substantially lower than that of Base.

Do the authors have any insights into these results?

Questions for authors

See above the weaknesses section.

Reviewer_AQ1eOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes E2T2S (Explicit-to-Tacit-to-Skill), a framework designed to extract reusable expert reasoning procedures from scholarly literature and leverage them to improve multimodal reasoning. The framework consists of three stages: (1) extracting structured explicit knowledge from scholarly papers, (2) aggregating evidence-use patterns across individual scholars' publications to infer tacit procedural knowledge, and (3) consolidating these patterns into an expert skill library that provides dynamically selected procedural guidance during inference.

The authors evaluate E2T2S on Chinese Chu bamboo-slip character interpretation, where a multimodal LLM must identify the correct modern Chinese character from four candidates using glyph images, textual context, external evidence, and scholarly literature. Using Qwen3-VL-8B-Instruct, E2T2S achieves 51.57% accuracy and 56.16% Macro-F1, outperforming the strongest reported baseline, RAG+CoT, by 10.54 and 14.55 percentage points, respectively.

Critical strengths

1. Well-motivated research problem.

The paper addresses an important distinction between accessing domain-specific knowledge and knowing how to effectively use that knowledge. In specialized reasoning tasks, retrieving relevant evidence does not necessarily guarantee that a model can appropriately select, integrate, or reconcile conflicting information. Exploring whether expert reasoning procedures can be recovered from scholarly literature is a meaningful research direction.

2. Potentially interesting scholar-centric approach.

Unlike conventional retrieval-based methods that primarily focus on accessing relevant information, E2T2S attempts to identify recurring reasoning strategies across a scholar's publications. The proposed representation of methodological preferences, decision templates, boundary conditions, and revision traces is conceptually interesting. In particular, the idea of extracting transferable procedures from accumulated scholarly practice, rather than individual documents, has potential value.

3. Encouraging empirical improvements.

The proposed method substantially outperforms the reported Base, CoT, RAG, and RAG+CoT baselines. The authors also attempt to control the retrieved scholarly evidence across relevant methods and provide evidence-aware evaluations and ablation studies. These experiments offer preliminary evidence that domain-specific procedural guidance can benefit the investigated task.

Critical weaknesses

1. Limited benchmark coverage and insufficient evidence of generalizability

The central motivation of E2T2S is that scholarly literature contains reusable tacit expert procedures that can improve specialized multimodal reasoning. This suggests a potentially general framework for transforming accumulated expert knowledge into transferable reasoning skills. However, the proposed framework is evaluated exclusively on Chinese Chu bamboo-slip character interpretation, formulated as a four-choice classification task. While this is a relevant and challenging application, I do not find it sufficient to establish the broader effectiveness of the proposed framework.

In particular, it remains unclear whether E2T2S learns generally useful procedures for evidence-based expert reasoning or primarily captures domain-specific heuristics that are particularly effective for ancient Chinese character interpretation. For example, several procedures illustrated in the paper, such as contextual semantic constraints, candidate elimination, and comparison against parallel texts, are closely aligned with the requirements of the evaluation task. Consequently, the reported improvements could potentially arise from task-specific procedural guidance rather than a broadly applicable mechanism for recovering tacit expert knowledge.

I believe evaluation on at least one additional task or domain would substantially strengthen the paper. Ideally, this would involve another specialized reasoning task requiring the integration of heterogeneous evidence from scholarly literature. An additional evaluation setting within ancient-text interpretation, using a different dataset or an open-ended interpretation task, would also provide useful evidence of robustness.

I recognize that a single-domain evaluation is not inherently insufficient for a research contribution. However, given the generality of the proposed E2T2S framework and its motivation, the current evidence supports a substantially narrower conclusion than the paper appears to suggest. Without further evaluation, the authors should more carefully scope their claims to the investigated paleographic task.

2. The core tacit knowledge extraction mechanism is insufficiently specified and validated

Section 3.2 defines scholar-level tacit knowledge as a combination of core stance vectors, decision templates, boundary and exception rules, and revision traces. However, the paper does not provide sufficient implementation details for the main extraction and construction procedures. In particular, several important questions remain unanswered:

  • How does the framework identify recurring decision patterns across a scholar's publications?
  • How are genuine methodological regularities distinguished from paper-specific observations?
  • How are revision traces identified and validated across different publications?
  • How are canonical skill patterns defined and aligned across scholars?
  • How is the evidence-state matching score computed for dynamic skill selection?

The corresponding functions, including the tacit extraction function, skill alignment function, and evidence-state construction function, are presented primarily as abstract mappings.

More importantly, the paper does not directly validate whether the extracted procedures faithfully represent the scholarly reasoning patterns from which they are supposedly derived. This is a critical issue because the main novelty is not simply the use of procedural instructions or dynamically selected skills. Prior work has already explored procedural knowledge, expert decision models, structured reasoning, and reusable skill libraries, including PlaSma (Brahman et al., 2024), Bridge (Wang et al., 2024), Voyager (Wang et al., 2023), and SkillWeaver (Zheng et al., 2025).

The distinctive contribution of E2T2S should therefore lie in systematically recovering reusable expert procedures from scholars' research trajectories. However, the paper does not convincingly distinguish this process from generating plausible domain-specific reasoning instructions based on scholarly texts.

I would encourage the authors to provide complete, provenance-traceable examples showing how multiple papers by the same scholar are transformed into explicit knowledge representations, recurring decision patterns, and final skills. An evaluation by domain experts of the fidelity and applicability of these extracted procedures would also strengthen the technical contribution.

3. The ablation study does not adequately isolate the contribution of each stage

Although the paper includes stage-wise ablations, their design makes the results difficult to interpret causally.

In Table 3, removing tacit knowledge extraction reduces accuracy from 51.57% to 27.11%, substantially below the Base accuracy of 38.74%.

However, according to Section 4.6, the w/o Tacit variant directly uses structured explicit knowledge for retrieval-based augmentation. This changes the representation and use of information provided to the model, rather than simply removing tacit procedural knowledge while keeping the remaining inference procedure unchanged. Consequently, the observed performance degradation could arise from ineffective knowledge presentation, irrelevant additional information, or other differences in the resulting prompts. It does not necessarily quantify the isolated contribution of tacit knowledge extraction.

Similarly, the w/o Explicit variant directly extracts tacit knowledge from scholarly papers and uses it as static procedural guidance. Since this ablation changes both the extraction pathway and the form of procedural guidance, it is difficult to attribute the performance decrease exclusively to explicit knowledge extraction.

I believe the authors should include more controlled ablations that preserve the same inference pipeline while modifying only the source or selection of skills. In particular, comparing single-paper versus scholar-level skill extraction and static versus dynamic skill selection would provide substantially stronger evidence for the proposed framework's key design choices.

Questions for authors
  1. Stronger and more diverse baselines. The current comparisons are limited to Base, CoT, RAG, and RAG+CoT, which may not be sufficient to establish the advantages of the proposed tacit knowledge extraction framework over existing procedural reasoning approaches. Could the authors provide comparisons with stronger baselines, such as expert-written procedural prompts, structured reasoning methods, and evidence-aware RAG approaches involving iterative retrieval or evidence verification? These comparisons would help clarify whether the improvements stem specifically from extracting tacit expert knowledge or simply from providing domain-specific reasoning guidance. Additionally, since all experiments use Qwen3-VL-8B-Instruct, it would be valuable to examine whether the improvements remain consistent across different multimodal backbones.

  2. Dataset split and evaluation statistics. Section 4.1 reports 644 instances with a 7:3 corpus/evaluation split. However, the sample counts across the seven evidence-aware conditions in Table 2 sum to exactly 644, while the first three conditions alone account for 268 instances. Could the authors clarify how these subsets relate to the held-out evaluation set, whether the conditions overlap, and whether all reported results are computed exclusively on held-out instances? Additionally, the Introduction refers to a paper–character double-isolation split, whereas Section 4.1 describes a scholar–character double-isolation split. The exact isolation strategy should be clarified. Appendix B also mentions three independent runs, but standard deviations are missing from the reported tables. Could the authors provide per-run results and explain the aggregation procedure, particularly for subset accuracies such as 48.33% on 137 Mixed Evidence instances? Finally, how is potential answer leakage from the scholarly evidence provided to evaluation instances prevented?

  3. Direct evaluation of reasoning capabilities. The paper's central motivation is that expert procedural knowledge improves how models select, integrate, and revise heterogeneous evidence. However, the current evaluation primarily measures final-answer accuracy and Macro-F1, which do not directly demonstrate improvements in these reasoning capabilities. Could the authors provide a more systematic evaluation of evidence selection, contradiction handling, hypothesis revision, and reasoning faithfulness, potentially through domain-expert annotations? Such an evaluation would help establish whether the observed performance gains reflect genuinely improved evidence-based reasoning rather than more effective final-answer selection. The qualitative examples in Appendix A are useful, but additional quantitative evidence would strengthen this claim.

  4. Generalization beyond the current task. Can the authors provide experimental evidence that E2T2S generalizes beyond the current four-choice Chu bamboo-slip interpretation setting, ideally through an additional task or domain requiring heterogeneous evidence integration? If not, could the authors clarify the intended scope of the framework's generalization claims?

  5. Tacit knowledge extraction and validation. Could the authors provide the complete extraction and skill construction procedures, including the prompts, intermediate representations, matching functions, and representative examples traced back to multiple publications by the same scholar? Has the quality or fidelity of the extracted tacit procedural knowledge been independently evaluated by domain experts?

  6. Ablation and skill selection. How do skills derived from individual papers compare with those obtained through scholar-level aggregation? Furthermore, how does dynamically selecting skills compare with using a fixed set of skills or randomly selected skills under comparable prompting and inference conditions?

Reviewer_gV9MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes E2T2S, a framework that extracts expert procedures from scholarly papers. It first extracts structured knowledge, then finds common methods across a scholar's papers, and finally builds an Expert Skill Library. At test time, the model selects relevant skills to guide its reasoning. The authors test the method on 644 cases of Chinese Chu bamboo-slip character interpretation. The task requires the model to use character images, text context, external evidence, and scholarly papers. The experiments use Qwen3-VL-8B-Instruct. E2T2S achieves 51.57% accuracy and 56.16% Macro-F1. It performs better than the tested Base, CoT, RAG, and RAG+CoT methods.

Critical strengths
  • The paper is well written, and the idea is interesting. It studies how models can learn expert methods from scholarly papers, rather than only retrieve facts.
  • The E2T2S framework has three clear stages. The Expert Skill Library also keeps links to the source papers. This makes the extracted skills easier to check.
Critical weaknesses
  • The paper may overstate the general value of the method. The authors present E2T2S as a general framework for learning expert skills from scholarly papers. However, the experiments focus on only one task: Chinese Chu bamboo-slip character interpretation. It is unclear whether the method can work in other fields, such as scientific research. The main claim should be more limited, or the authors should test the method on more tasks.

  • The source of the performance gains is not clear enough. In Lines 414–417, the authors link the gains in Table 1 to the use of procedural knowledge. However, the current experiments do not directly prove this claim. The gains may also come from better skill instructions or a better prompt design. More control tests are needed to separate these effects.

  • The analysis of evidence use is limited. The paper mainly reports accuracy, Macro-F1, ablation results, and two case studies. However, the method aims to improve evidence selection, integration, and revision. The authors should provide more visual examples and deeper analysis. For example, they should check whether the model truly uses relevant evidence, whether it can handle conflicting evidence, and what happens when the wrong skills are selected.

  • The model comparison is limited. All experiments use Qwen3-VL-8B-Instruct. The paper does not compare the method with strong open or closed models, such as GPT-6. It is unclear how well these models perform on this task without expert skills. It is also unclear whether E2T2S can further improve their results.

Questions for authors
  • Can the authors test E2T2S on other tasks, such as scientific research? How can they show that the method works beyond ancient Chinese character interpretation?

  • Can the authors compare E2T2S with a control method that uses prompts of similar length but does not use the extracted expert procedures? This would help show whether the gains come from procedural knowledge or better prompts.

  • How do the authors check whether the model truly uses the selected evidence? Can they provide more cases with conflicting evidence and show how the model changes its decisions?

  • What happens when the skill selection step retrieves irrelevant or incorrect skills? How much does this affect the final results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.