acceptodds
Under review as a conference paper at ICLR 2027

The Grid as Observable State: Infrastructure-Aware Resilience for Large-Scale Training

Abstract

Large-scale distributed training relies on checkpointing to recover from failures. Its resilience also depends on the infrastructure shared by the sites that host it. We argue that resilience evaluations should make this shared exposure visible alongside hardware failure statistics. Using five years of hourly ERCOT electricity-price data, we characterize the spatial and temporal structure of negative-price episodes. In 2025, negative-price indicators at the studied North Texas substations have pairwise correlations of 0.875–0.933. In the corridor-level series, episodes last 3.8 hours on average, and 46 of 56 episode starts (82.1%) fall between 03:00 and 12:00 CST, a window covering 37.5% of the day. These observations reveal shared market exposure and concentrated timing, two dimensions of infrastructure context relevant to resilience evaluation. Whether market conditions affect training availability depends on the facility's operational response. Under an explicit episode-to-interruption scenario, a checkpointing illustration connects the observed timing to a break-even criterion involving interruption exposure, checkpoint effort, and recovery costs. Comparisons at matched nominal checkpoint frequency show why resource accounting matters when assessing adaptation. Together, the empirical findings and conditional analysis motivate evaluating observable infrastructure signals alongside facility telemetry to identify when they can improve training-resilience decisions.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Distributed Training, Fault Tolerance, Infrastructure-Aware Resilience, Checkpointing, Electricity Markets

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_3bdjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether external grid conditions can provide useful information for improving the resilience of large-scale AI training. Using five years of ERCOT real-time electricity-price data, the authors characterize the spatial and temporal structure of negative-LMP events, including correlations across selected substations, persistence of negative-price episodes, and concentration of episode starts during certain hours of the day. Under an explicit episode-to-interruption scenario, a checkpointing illustration connects the observed timing to a break-even criterion involving interruption exposure, checkpoint effort, and recovery costs.

Critical strengths

The paper is generally clearly written and presents a motivating perspective that external infrastructure conditions may introduce correlated and time-varying risks to large-scale training systems.

Critical weaknesses

The main concern is that the paper does not establish a clear connection between the proposed grid signal and actual training interruptions. Although resilience is emphasized in both the title and the paper, it remains unclear how the observed grid conditions actually affect training resilience. Negative LMP is an electricity-market signal and does not necessarily indicate power shortage, curtailment, or interruption, which the paper itself also acknowledges. However, the checkpointing analysis assumes that negative-LMP episodes can lead to training interruptions. As a result, the empirical analysis mainly demonstrates that electricity prices exhibit spatial and temporal structure, but it remains unclear whether this structure is relevant to training failures. This missing connection significantly weakens the main motivation and conclusions of the paper.

Questions for authors
  1. Do the authors have any empirical evidence showing that negative-LMP events are correlated with actual data-center curtailments, power interruptions, or other events that could interrupt training? Establishing this connection seems important for understanding whether the proposed grid state is actually informative for resilience decisions.

  2. To my understanding, data centers would generally not interrupt operation simply because electricity prices become unfavorable, particularly given the high value of ongoing training workloads relative to electricity costs. Moreover, negative electricity prices often reflect sufficient renewable energy resources, rather than an increased risk of power loss. Could the authors clarify the physical or operational mechanism through which negative-LMP events would increase the risk of training interruption?

  3. It would also be helpful to clarify what exactly is meant by 'resilience' in this setting and how its impact is quantified. For example, when an interruption occurs, is training assumed to stop immediately and later resume from the most recent checkpoint? What are the resulting costs or consequences of an interruption, and how are they incorporated into the resilience metric?

  4. Can the authors evaluate the proposed method out of sample? For example, can the high-risk time window or risk profile be estimated using data from 2021--2024 and then evaluated on 2025? It would also be useful to compare a time-of-day-only policy with a policy that additionally uses the grid state, to quantify whether the grid observation provides incremental predictive value.

Reviewer_bHr4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that resilience studies of large-scale training should report exposure to shared infrastructure, in particular the power grid.

Critical strengths

The paper is clear about its limits. It states that a negative price is not an outage and that interruptions are not observed.

Critical weaknesses
  1. There is no link between the data and training interruptions. The quantity the paper itself defines as relevant, $ P(I_{t+1}=1 \mid G_t, h_t, X_t) $, is never estimated, because no interruption, curtailment, or job data are used. The link from price to interruption is simply assumed in Section 4. The assumption is also hard to accept on its face. A negative price means there is more power than the local market needs, and the paper notes that it does not mean power is unavailable. Grid risk to a large load comes from scarcity, which goes with high prices, not negative ones. A data center seeing negative prices has no reason to stop. As a result, Section 3 describes ERCOT price behavior, not anything about training.

  2. Even under the authors' assumption, the gain is very small. The time-of-day rule writes about twice as many checkpoints as the 4 h baseline (23/48 vs. 1/4 per hour). When the number of writes is matched, the rollback saving drops to 9.3%, and to 1.4% (0.76 h per year) once the extra daily write is counted. The paper never sets the checkpoint cost rr, so Eq. (14) is never evaluated. If 4 h is taken as the Young interval for λ=56/8760\lambda = 56/8760, then r≈3r \approx 3 min. In that case the extra writes cost about 103 h per year against a 59 h saving, so total overhead goes up.

  3. The paper has no learning component, which makes it a weak fit for ICLR. This point did not drive my score.

Questions for authors

Is there any evidence, from a facility, an ERCOT program, or a reported incident, that negative real-time prices have preceded a data-center or training interruption? If not, why not use a scarcity signal instead?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.