- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses a well-defined and practically relevant problem established by closely related work (GREATS, OPUS). The paper's main contribution is to introduce a curvature estimate with retained directional information to address the gap that was left open by related work. Furthermore, the paper introduces second-order methods to stabilize the training process and improve evaluation results. Experimental results cover pre-training and fine-tuning.
Critical strengths
The paper addresses a well-defined and practically relevant problem established by closely related work (GREATS, OPUS). The idea of introducing a curvature estimate with retained directional information can help alleviate what related work left open before. The idea of using second order methods is cheap and scales in practice, which is helpful as model parameter sizes grow. Showing applicability to both pre-training and fine-tuning is helpful for practical guidance.
Critical weaknesses
The paper has several areas of improvement. I list them by impact:
- None of the experimental results contains a standard deviation or confidence interval. There is no evidence that the results reported in the paper are a product of chance or whether DRESS is reproducible. I understand that pre-training is cost and resource intensive but for a study like this having a standard deviation across at least 3 seeds is a minimum, I think.
- Almost all pre-training benchmarks show results at near chance (e.g., MMLU reports 25-33%; with random choice being 25%). At such low benchmark scores it tells little about the performance of DRESS. Strangely, ARC Challenge is below chance (without an explanation why). This hints at two things that warrant a more thorough discussion and analysis, potentially involving larger models (GPT-2-style models are almost 8 years old).
- While the paper discusses some sample-level reweighting approach, it does not provide an experimental comparison. This comparison could provide the reader a more fundamental understanding how the proposed DRESS method compared against existing work.
- There are several issues on the benchmark and evaluation design: The paper seems to copy baseline numbers from the OPUS paper instead of running the method on their own to ensure an apples-to-apples comparison. Hence, the insights discussed only have limited value. Similarly, the value appears to be tuned on the target benchmarks, without providing a discussion on how to tune for practical settings. Also for pre-training, it seems the results for Muon are only theoretically derived but never evaluated experimentally (only AdamW on the LM heads). Furthermore, the setup of lora experiments is very narrow (1 dataset only with a base model that performs strongly on the target benchmark).
Questions for authors
- What would be a practical intuition for tuning ?
- How does DRESS compare to closely related sample-level reweighting approaches like LinUpper or InfoBatch?
- What is the end-to-end FLOP comparison including the forward passes on the 60B candidate tokens? How do the results compare to the Random baseline at identical wall-clock budget?