acceptodds
Under review as a conference paper at ICLR 2027

Built to Read, Not to Recall: Language Skills over World Knowledge in Small Language Models

13 views

Abstract

Frontier-scale language models are commonly assumed to need their full scale. We argue instead that LLM abilities fall into two families with very different scaling behavior: world knowledge — stored facts and entities — improves steeply with parameter count, whereas language skills — comprehending, extracting, and structuring given text — are largely attainable at modest sizes when cultivated deliberately. Retrieval-augmented generation (RAG) and agent pipelines keep knowledge external and rely chiefly on the latter; a compact purpose-trained model can therefore replace frontier-scale components at a fraction of the cost and energy. We test this hypothesis with Meno-Lite-0.1, a publicly released bilingual Russian–English 7B-class model whose name inverts Plato's anamnesis: the knowledge lives in the corpus, the skill in the model — built to read, not to recall. Across a 0.5B–72B backbone size ladder, knowledge benchmarks climb steeply (closed-book quiz F1 grows 21× from 0.5B to 72B) while skill benchmarks plateau after  7B, and the memorization ratio — answering from parameters instead of the given context — rises with scale. The skill-trained 7B model sits above the entire ladder on knowledge-graph construction (0.468 vs. 0.416 harmonic mean against a 32B model), reaches 92% of the 32B level on open-book multi-hop QA while retaining its backbone's knowledge level, is the most context-faithful model measured, and serves Russian text 6.2× faster than the 32B engine. We further document protocol sensitivities that can masquerade as scaling behavior. Weights, training data, and code are publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
small language models, retrieval-augmented generation, tool calling, structured output generation, scaling laws, knowledge graph construction, tokenizer adaptation, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RYeYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

A distinction between parametric world knowledge and context-grounded language skills is investigated. Meno-Lite-0.1, a bilingual 7B model, is developed through skill-oriented continued pretraining and supervised fine-tuning. Knowledge, extraction, question answering, and context faithfulness are evaluated across a Qwen2.5 size ladder and Russian-adapted peers. Strong extraction performance and improved context adherence are reported, alongside analyses of evaluation protocols and serving efficiency.

Critical strengths
  1. A practically relevant question is examined: how much model capacity is required when task knowledge is supplied through context. Knowledge benchmarks, context-grounded tasks, and counterfactual-context evaluation are brought together in a useful experimental framework.
  2. Substantial extraction improvements are demonstrated. On NEREL-bench, a harmonic mean of 0.4676 is obtained, compared with 0.3558 for Qwen2.5-7B and 0.4163 for Qwen2.5-32B. Independent evidence of improved context use is also provided through MultiQ, where F1 increases from the 7B backbone’s 0.425 to 0.536.
  3. Evaluation artifacts are investigated through controlled experiments. Chat-template effects, answer verbosity, and extraction refusals are examined using raw generations. Serving efficiency is measured under a common hardware and software configuration, with Russian throughput benefits traced to both model size and tokenizer density.
Critical weaknesses
  1. The saturation claim is broader than the controlled evidence. Between 7B and 32B, MultiQ F1 increases from 0.425 to 0.585 and NEREL harmonic mean increases from 0.356 to 0.416. These results indicate task-dependent scaling. The 1.5B model provides partial evidence for transferring the recipe across sizes, while a controlled comparison of skill-trained models at larger scales would more directly test the proposed 7B saturation boundary.
  2. The strongest extraction result is closely aligned with the training distribution. NEREL-derived instructions are used during SFT, and the evaluation shares its annotation schema and domain. Separate training and test splits are used, so the concern concerns transfer across schemas and domains. Evaluation on independent extraction datasets would strengthen the general-skills interpretation. The breadth of bilingual transfer would also be clarified by evaluating Meno-Lite on HotpotQA and MMLU-Pro.
  3. Several interpretations conflict with the reported tables. Section 5.3 states that adapted peers fall below the 32B model on extraction, whereas Table 4 gives YandexGPT-5-Lite 0.379 and Qwen2.5-32B 0.355 under the same protocol. Section 5.4 states that monotonicity is recovered after refusals are excluded, whereas Appendix D retains reversals at 3B and 32B. These statements should be corrected, with the scaling interpretation updated accordingly.
Questions for authors
  1. What scaling pattern is obtained when the same skill-training recipe is applied to 7B, 14B, and 32B backbones under controlled data budgets?
  2. How much of the extraction improvement is retained on an independent domain and annotation schema?
  3. Can unconditional counterfactual-context accuracy, original-fact response frequency, and neither-fact frequency be reported for every model?
Reviewer_WaKsOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • The paper is based on the following hypothesis: "LLM abilities fall into two families with very different scaling behavior: world knowledge—stored facts and entities—improves steeply with parameter count, whereas language skills comprehending, extracting, and structuring given text are largely attainable at modest sizes when cultivated deliberately."
Critical strengths
  • The paper addresses a problem involving large scale language models
Critical weaknesses
  • The paper suffers readability issues
  • The whole paper is not clear, from the introduction to the conclusion.
  • Many sentences are excessively long, making the paper difficult to read and understand
  • Several important terminologies are not defined before first use (e.g., GraphRAG)
  • The related work section mixes existing work with the paper's contributions, making it difficult to distinguish prior research from the authors' contributions
  • Table 1 is difficult to interpret because the correspondence between mismatched elements across columns is unclear
  • The research methodology is insufficiently presented. A research methodology should clearly explain the research design, proposed approach. Presenting two tables with difficult to follow explanations does not adequately establish how the research was conducted.
Questions for authors
  • What is the motivation of the paper?
  • Which problem the paper is supposed to solve?
Reviewer_hjafOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether language skills and world knowledge exhibit different scaling behaviors in language models. The authors argue that abilities such as reading comprehension, information extraction, and reasoning over provided context can be developed effectively in smaller models, while parametric world knowledge benefits more substantially from increasing model size. To study this, they evaluate models across a Qwen2.5 size ladder and introduce Meno-Lite-0.1, a 7B-class model trained for context-grounded tasks.

I find the research question interesting and relevant, particularly for RAG and agent-based systems where much of the required knowledge is already available in external sources. The paper is generally well written, and the experimental analysis covers several aspects, including knowledge benchmarks, language-oriented tasks, context faithfulness, and serving efficiency. I also appreciate the discussion of evaluation protocol sensitivity. The examples showing how chat templates and response formatting can substantially change benchmark results are useful and highlight an important issue in comparing language models.

However, I have concerns about whether the experimental evidence fully supports the paper's central hypothesis. The distinction between world knowledge and language skills is reasonable as a working hypothesis, but the selected benchmarks do not necessarily provide a clean separation between these capabilities. Some language-skill benchmarks still improve substantially from 7B to 32B, while several important evaluations are missing for the largest models. This makes the claim that language skills generally plateau around 7B less convincing.

I am also concerned about the interpretation of Meno-Lite's improvements. The strongest result is obtained on NEREL-bench, but the model was trained using instructions derived from the same annotation framework and domain. Although this does not necessarily indicate data leakage, it makes it difficult to distinguish broader language-skill improvements from task-specific adaptation. Additional evaluation on independent extraction benchmarks and more controlled training ablations would help strengthen this conclusion.

Finally, the paper builds on closely related work that has already demonstrated the benefits of compact, context-oriented models, including the previously reported Meno-Lite results. I would like to see a clearer explanation of what new understanding this submission provides beyond those findings.

Overall, I think the paper asks a worthwhile question and contains useful empirical observations, especially regarding evaluation protocol sensitivity. However, I am not fully convinced that the current experiments establish the proposed scaling distinction or sufficiently isolate the contribution of the skill-oriented training strategy. I therefore lean toward rejection in its current form.

Critical strengths

1. Interesting research question with practical implications. The paper investigates whether compact language models can provide the capabilities needed for retrieval-grounded systems without requiring the parametric knowledge of much larger models. I find this question valuable because it could influence how language models are trained and selected for RAG and other knowledge-intensive applications. The strong extraction performance of Meno-Lite provides encouraging evidence that targeted training can make smaller models competitive in specialized tasks.

2. Useful analysis of evaluation protocol sensitivity. I particularly appreciate the experiments showing how chat templates and output formatting affect generative benchmark scores. The reported changes are substantial, and the analysis demonstrates that apparent differences between model sizes may sometimes reflect evaluation artifacts rather than underlying capabilities. I consider this one of the more informative contributions of the paper.

Critical weaknesses

1. The evidence for the proposed scaling distinction is not sufficiently conclusive. The paper argues that language skills plateau around 7B while world knowledge continues to improve with model size. However, the results are more mixed than this conclusion suggests. For example, MultiQ F1 increases from 0.425 at 7B to 0.585 at 32B, which is still a substantial improvement. Several relevant benchmarks are also missing for the 72B model, and the analysis is restricted to one backbone family. Moreover, the distinction between knowledge and language skills is not fully isolated by the selected tasks. These limitations weaken the paper's central claim about the different scaling behaviors of the two capability families.

2. Limited isolation of the contribution of skill-oriented training. Meno-Lite achieves its strongest improvement on NEREL-bench, but its supervised training includes instructions derived from the same annotation framework and domain. While the paper acknowledges this limitation, the results may partly reflect task-specific adaptation rather than a more general improvement in language skills. The lack of separate continued-pretraining and supervised-fine-tuning ablations also makes it difficult to determine which parts of the training recipe are responsible for the observed gains. In addition, closely related prior work has already reported the model's main extraction advantage, leaving the independent contribution of the present submission somewhat unclear.

Questions for authors

1. Could the authors clarify the evidence supporting the claim that language skills plateau around 7B? Table 3 shows a substantial improvement in MultiQ performance between 7B and 32B, and several language-skill benchmarks are not evaluated at 72B. How sensitive is the proposed distinction to the selection of benchmarks, particularly when excluding tasks affected by output-format artifacts? This would help me assess whether the reported trend reflects a general scaling property or the particular tasks included in the study.

2. How much of Meno-Lite's improvement can be attributed to general language-skill training rather than task-specific adaptation? Since the model uses NEREL-derived instructions during SFT, I would be interested in seeing results on an independent information-extraction benchmark with a different annotation schema and domain. Could the authors also clarify the respective contributions of continued pretraining and supervised fine-tuning? These results would materially affect my interpretation of the model's performance gains.

3. Could the authors clarify the main new contribution relative to the earlier RAGU and OCC-RAG findings? The previously reported work already motivates compact models for context-grounded tasks, and RAGU reports the main Meno-Lite extraction result. I would like to better understand which conclusions in the current paper are new and independently supported by the additional scaling and protocol-sensitivity experiments. This distinction is important for assessing the paper's methodological and empirical novelty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.