- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
A distinction between parametric world knowledge and context-grounded language skills is investigated. Meno-Lite-0.1, a bilingual 7B model, is developed through skill-oriented continued pretraining and supervised fine-tuning. Knowledge, extraction, question answering, and context faithfulness are evaluated across a Qwen2.5 size ladder and Russian-adapted peers. Strong extraction performance and improved context adherence are reported, alongside analyses of evaluation protocols and serving efficiency.
Critical strengths
- A practically relevant question is examined: how much model capacity is required when task knowledge is supplied through context. Knowledge benchmarks, context-grounded tasks, and counterfactual-context evaluation are brought together in a useful experimental framework.
- Substantial extraction improvements are demonstrated. On NEREL-bench, a harmonic mean of 0.4676 is obtained, compared with 0.3558 for Qwen2.5-7B and 0.4163 for Qwen2.5-32B. Independent evidence of improved context use is also provided through MultiQ, where F1 increases from the 7B backbone’s 0.425 to 0.536.
- Evaluation artifacts are investigated through controlled experiments. Chat-template effects, answer verbosity, and extraction refusals are examined using raw generations. Serving efficiency is measured under a common hardware and software configuration, with Russian throughput benefits traced to both model size and tokenizer density.
Critical weaknesses
- The saturation claim is broader than the controlled evidence. Between 7B and 32B, MultiQ F1 increases from 0.425 to 0.585 and NEREL harmonic mean increases from 0.356 to 0.416. These results indicate task-dependent scaling. The 1.5B model provides partial evidence for transferring the recipe across sizes, while a controlled comparison of skill-trained models at larger scales would more directly test the proposed 7B saturation boundary.
- The strongest extraction result is closely aligned with the training distribution. NEREL-derived instructions are used during SFT, and the evaluation shares its annotation schema and domain. Separate training and test splits are used, so the concern concerns transfer across schemas and domains. Evaluation on independent extraction datasets would strengthen the general-skills interpretation. The breadth of bilingual transfer would also be clarified by evaluating Meno-Lite on HotpotQA and MMLU-Pro.
- Several interpretations conflict with the reported tables. Section 5.3 states that adapted peers fall below the 32B model on extraction, whereas Table 4 gives YandexGPT-5-Lite 0.379 and Qwen2.5-32B 0.355 under the same protocol. Section 5.4 states that monotonicity is recovered after refusals are excluded, whereas Appendix D retains reversals at 3B and 32B. These statements should be corrected, with the scaling interpretation updated accordingly.
Questions for authors
- What scaling pattern is obtained when the same skill-training recipe is applied to 7B, 14B, and 32B backbones under controlled data budgets?
- How much of the extraction improvement is retained on an independent domain and annotation schema?
- Can unconditional counterfactual-context accuracy, original-fact response frequency, and neither-fact frequency be reported for every model?