acceptodds
Withdrawn from ICLR 2027

As a Hidden Hyperparameter, Hardware Changes the Training Algorithm

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Splitting a training update across GPUs or microbatches should save time and memory, not change the algorithm. Yet in practice, it can. We introduce Partition Invariance: repartitioning the same data must preserve the training update in exact arithmetic when the specified algorithm is unchanged. We prove a sufficient condition for this property and use its premises to guide diagnosis. PartitionCheck tests this property by comparing the same update across two partition configurations, allowing for floating-point differences. It also checks each execution against an independent reference implementation, since matching executions can share the same error. Beyond detection, we characterize how partitioning changes the algorithm. Under stated conditions, constant gradient scaling changes the effective learning rate under SGD but weight decay under LARS. In five codebases, we test whether code-derived rewrites reproduce the affected runs. Across 31 frameworks and research codebases, including Transformers, DeepSpeed, and veRL, we identify 43 implementation errors; six affect every partition alike. These errors can undermine research conclusions even without degrading model quality. In SimCSE, two GPUs double the relative weight of an auxiliary loss. Fixed padding bypasses it, making an ablation compare identical training runs. In a preregistered experiment with a 134M-parameter Mamba-2 model, the affected tensor-parallel implementation yields 0.189 nats higher test loss than its repaired counterpart when both are evaluated under the declared architecture. Reporting the partition is not bookkeeping; it is part of reporting the method.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
training correctness, distributed training, partition invariance, machine learning systems, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P3f1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper addresses the problem of how different number of GPUs, micro batch allocation, padding and gradient accumulation can influence the training updates even when all the other hyper-paramters are kept the same. In particular, they propose PartitionCheck, which can test if two training configurations, differing only in terms of the aforementioned properties, provides similar updates, up to floating-point tolerance. Notably, PartitionCheck can be used to also compare the two runs to a third independent reference run. The authors inspected various code implementations for model training (.e.g, the ablation in SimCSE), finding bugs, and they used PartitionCheck to confirm these bugs.

Critical strengths
  1. The paper tests PartitionCheck on real code from widely used libraries( e.g., Transformers). These tests lead to confirmation of real bugs that were communicated to the code owners, proving that PartitionCheck can work on real code.
  2. Some of these bugs impacted the research findings; for example in SimCSE, the extra MLM loss was wrongly computed depending on the number of GPUs. This has been verified using PartitionCheck against a reference run.
Critical weaknesses
  1. The proposed PartitionCheck algorithm overlaps with the TTrace paper. In particular, the authors say that “PartitionCheck therefore reports reference conformance separately from partition consistency.” [Section 3.3]; one could use TTrace by comparing the two systems separately against a reference, and the comparing the results, thus obtaining similar information. Given so, it is unclear how much novelty PartitionCheck adds with respect to TTrace.
  2. The text is really hard to read in many places: “The price ranged from no detectable change in Mask2Former’s panoptic quality to 0.189 nats of test loss in four-way Mamba-2 under declared-architecture evaluation,”[Conclusions], “passed scoped dual-host repair closure”Section 4.1, line 347], “under the declared architecture” [Abstract], “One declared algorithm, several executed algorithms” [Figure 1], “Each panel varies only a resource knob” [Figure 1], “frozen tolerances” [Figure 2], “Covered operations” [Figure 2 and in other places], “supported partition plans p and q” i[Section 3.1, line 360], …
Questions for authors
  1. Can you specify in detail the difference between TTrace and PartitionCheck, focusing on what can be done with the latter but not with the former?
  2. Would it be possible to clarify the main hard-to-read points in the paper, providing a more natural reading experience?
Reviewer_EgZZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how changes in GPU count or microbatch partitioning can unintentionally alter training semantics. It formalizes partition invariance and introduces PartitionCheck, which compares executions intended to implement the same logical update, checks intermediate quantities and resulting states, and uses reference computations to diagnose discrepancies. Its empirical contribution is a collection of implementation audits and controlled case studies.

Critical strengths
  1. The paper addresses a practically consequential reproducibility problem: changing execution partitioning can silently change the implemented algorithm while training still appears to proceed normally. The case studies illustrate why successful convergence alone is insufficient evidence of implementation correctness.

  2. Several investigations connect observed discrepancies to concrete implementation mechanisms and use repairs or controlled comparisons to examine their consequences. This provides stronger evidence than merely reporting numerical differences between execution plans.

Critical weaknesses
  1. The methodological contribution beyond established consistency testing, differential testing, and reference-implementation comparisons is insufficiently clear. Formalizing the requirement that equivalent execution plans preserve training semantics is useful, but the paper does not convincingly establish what substantially new detection capability follows from this formalization. The contribution is primarily a collection of systematic audits, and I find the additional generalizable insight insufficient for acceptance.

  2. The evaluation does not adequately establish reliability and practical applicability on unfamiliar codebases. The selected cases and injected faults demonstrate that the procedure can diagnose the tested discrepancies, but provide limited evidence about subtle-error detection, false alarms, missed faults, or the effort required to construct adapters, specify contracts, and build independent references. These dependencies limit the evidence for broadly applicable detection with modest manual effort.

Questions for authors
  1. What detection or diagnostic capability does PartitionCheck provide beyond applying existing differential or metamorphic testing to partition-equivalent executions? Please identify the specific technical contribution responsible for that capability.

  2. What evidence is available on previously unaudited codebases regarding adapter and reference-development effort, false alarms, and detection of smaller semantic deviations near numerical tolerances?

Reviewer_NrWKOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the effect of partitioning in the distributed ML training, in particular, cases when partitioning changes the result of an algorithm. It formalizes the conditions for an algorithm to be partition invariant, report relevant implementation errors across public codebases and proposes instructions to verify partition invariance.

This paper has meaningful practical contribution but at a current stage it is more of a engineering report than a scientific study.

Critical strengths
  1. Practical relevance. The paper reports bugs and proposes mitigations for distributed learning algorithms, which are widely used nowadays.
Critical weaknesses
  1. Limited novelty. The main contribution of the paper falls under engineering domain: it's a framework to verify the result of an algorithm does not change with the number of workers. While very practically relevant, the paper does not provide new scientific insight.

  2. The proposed PartitionCheck is not sufficiently evaluated. As I understood the majority of tests were training trace modifications, not the algorithm modifications themselves. It would be reasonable to expect it to find at least some of the 43 reported bugs across public codebases.

Questions for authors
  1. Has PartitionCheck found a previously undetected partitioning related errors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.