- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a method with some rules to "repair" low quality data so that they can become more useful in LLM pre-training. A set of heuristic rules for such repair has been proposed, following a pipelined framework shown in Figure 2. The proposed method has then been evaluated in pretraining a Qwen3-1.7B model with 34.4B tokens. The results show some improvement in the overall performance when using the proposed repair method compared to without it.
Critical strengths
- The paper could be of practical relevance for teams who pretrain LLMs.
- The description is detailed enough to represent the proposed approach.
- The experiment includes pretraining a real model.
Critical weaknesses
- The paper is poorly structured. For example, RQ1 appears in its own section while RQ2 and RQ3 appear as part of the results.
- The paper is mostly an engineering description. Some important discussions such as why the algorithm is designed in this way instead of some other way are essential for research papers but are largely missing.
- The proposed approach may not be difficult to come up with by an engineer experienced in pretraining. Some aspects of it may also already be included in text extractors (or could be easily included there) used in practice.
- Even from an engineering perspective, important aspects such as the compute it requires to run the repair pipeline are missing from the paper. This is important since such data processors need to be very lightweight to be able to process the huge amount of data that is used for training practical LLMs, which the current experiment with the 1.7B parameter model and 34.4B tokens may not be facing.
Questions for authors
- Could you improve the structure of the paper?
- Can you explain the core principles and reasons behind the design, including why these rules/algorithms instead of others?
- What is the novel contribution that is non-obvious for a hands on person?
- How is the compute consumption of this method compared to commonly used data processing pipelines that filter out bad quality data?