- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper addresses the problem of how different number of GPUs, micro batch allocation, padding and gradient accumulation can influence the training updates even when all the other hyper-paramters are kept the same. In particular, they propose PartitionCheck, which can test if two training configurations, differing only in terms of the aforementioned properties, provides similar updates, up to floating-point tolerance. Notably, PartitionCheck can be used to also compare the two runs to a third independent reference run. The authors inspected various code implementations for model training (.e.g, the ablation in SimCSE), finding bugs, and they used PartitionCheck to confirm these bugs.
Critical strengths
- The paper tests PartitionCheck on real code from widely used libraries( e.g., Transformers). These tests lead to confirmation of real bugs that were communicated to the code owners, proving that PartitionCheck can work on real code.
- Some of these bugs impacted the research findings; for example in SimCSE, the extra MLM loss was wrongly computed depending on the number of GPUs. This has been verified using PartitionCheck against a reference run.
Critical weaknesses
- The proposed PartitionCheck algorithm overlaps with the TTrace paper. In particular, the authors say that “PartitionCheck therefore reports reference conformance separately from partition consistency.” [Section 3.3]; one could use TTrace by comparing the two systems separately against a reference, and the comparing the results, thus obtaining similar information. Given so, it is unclear how much novelty PartitionCheck adds with respect to TTrace.
- The text is really hard to read in many places: “The price ranged from no detectable change in Mask2Former’s panoptic quality to 0.189 nats of test loss in four-way Mamba-2 under declared-architecture evaluation,”[Conclusions], “passed scoped dual-host repair closure”Section 4.1, line 347], “under the declared architecture” [Abstract], “One declared algorithm, several executed algorithms” [Figure 1], “Each panel varies only a resource knob” [Figure 1], “frozen tolerances” [Figure 2], “Covered operations” [Figure 2 and in other places], “supported partition plans p and q” i[Section 3.1, line 360], …
Questions for authors
- Can you specify in detail the difference between TTrace and PartitionCheck, focusing on what can be done with the latter but not with the former?
- Would it be possible to clarify the main hard-to-read points in the paper, providing a more natural reading experience?