- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces a monitor that detects training failures. This monitor has two applications: FlashAttention, and MoE routers. This is a three-step process: first, identify the persistent affect of the fault; second, derive the structural change; third, construct a monitor.
Critical strengths
The paper addresses the important problem of error diagnosis based on internal states. We need failure diagnosis methods beyond downstream task evaluation metrics.
FlashAttention and MoE models are widely used in nowadays AI/LLM systems.
Critical weaknesses
-
Limited experiment scale, and lack of experiment details: MoE experiments only use Qwen3 8B-A1B and no other models. I did not see any detailed experiment setup for the FlashAttention part.
-
A unified method is unclear/missing: I did not see any shared diagnosis method/pipeline proposed. FA and MoE are separate problems that require separate attention. What is the connection between them? Is the method proposed only to solve these problems? The paper looks more like two independent case studies than a unified method.
-
Limited comparison/discussion with prior diagnosis approaches. What is the unique contribution of this paper, compared to existing approaches?
-
I fail to see any concrete empirical validations (e.g., improved training performance, measured by task metrics) that can show why the proposed method is superior, or should be used.
Questions for authors
Please kindly refer to weaknesses.
7 comments
Weakness 2: Unified Method and Its Two Instances
Thank you for asking about the unified method and the connection between the two instances. Section 3 presents a method for constructing monitoring representations from fault-propagation mechanisms; FA and MoE instantiate this method in different modules. Given a target mechanism and the affected module, the method identifies the internal states that are persistently affected, derives the expected structure and constructs a monitoring representation based on the module's computation. Both instances follow this shared process, as detailed in Rebuttal Table 2.
Rebuttal Table 2: The shared mechanism-aligned design process for FA and the MoE router
| Shared design step | FA | MoE router |
|---|---|---|
| Identify the target mechanism | Biased errors from low-precision backward computation repeatedly enter Q/K updates | LR, GBS, and AUX affect routing dynamics through relative expert-score margins |
| Locate the persistently affected state | Recent Q/K parameter updates | Router parameters and the relative expert-score margins they induce |
| Derive the expected structure | Under conditions including a low-rank mean and controlled fluctuations, accumulated errors can concentrate the update spectrum | Feedback that reinforces the current winning experts can drive routing probabilities toward concentration and saturation |
| Identify the mismatch in existing representations | The historical component in raw weights can obscure recent update structure | Changes in router weight similarity need not correspond to changes in the routing function |
| Construct a function-aligned representation | Move from to windowed updates , then use the bilinear QK computation to construct the functional increment | Move from parameter similarity to per-token routing entropy to track probability concentration and saturation |
The mechanism analysis directly determines the representation in each instance. This unified method can also be applied to other training modules whose fault-propagation paths and computational structure can be analyzed. The monitoring quantity is derived from the target module's mechanism, rather than prescribing a single metric for every module. We will clarify this in the revision.
Weakness 3: Comparison with Prior Diagnostic Approaches
Our paper already includes related-work discussion and targeted comparisons of monitoring representations. Our distinct contribution is to derive these representations from mechanism analysis and evaluate the resulting choices in actual training. Rebuttal Table 3 summarizes our specific contributions relative to these diagnostic approaches and mechanism studies.
Rebuttal Table 3: Comparison with existing diagnostic approaches and our contributions
| Existing work | Our specific contribution |
|---|---|
| Global or parameter-level statistics, such as loss, gradient norm, and raw weights | Analyze why these representations can obscure mechanism-related changes, and compare windowed parameter updates with functional updates to demonstrate how representation choice affects fault observability |
| Qiu & Yao's analysis of biased errors in low-precision FA | Further analyze error accumulation within update windows, derive the predicted spectral concentration under the corresponding conditions, and construct the functional increment and its decomposition aligned with the bilinear QK computation |
| Behavior-level monitoring, such as attention logits and max-logit | Provide a functional-update representation based on periodic Q/K parameter snapshots, without modifying the fused FA kernel or collecting additional attention activations; compute the nonzero singular spectrum exactly using a small core obtained by QR factorization |
| Router weight-similarity monitoring | Use functional invariance to show that parameter similarity does not uniquely characterize the routing function, then derive the choice of per-token routing entropy from relative expert-score margins to monitor probability concentration and saturation |
Thank you for the insightful comments! We are sorry that our paper presentation did not meet your satisfaction. We have revised the paper based on your suggestions and those of other reviewers, hoping that it and the following discussion of the weaknesses will address your concerns.
Thank you for asking about model coverage. We clarify the rationale for the primary architecture and the additional evidence for the FA and Router studies.
We chose Qwen3 8B-A1B MoE for the main training experiments because its 28 layers, 128 routed experts, and top-6 sparse selection per token retain the key structures examined in our analysis: competition between expert scores, concentration of routing probabilities, and load-balancing feedback. These structures correspond directly to the mechanism analysis in Section 3.2. Varying LR, GBS, and AUX within a fixed architecture allows us to compare their effects on routing dynamics while reducing confounding changes in model size, expert count, and routing scheme. Compared with the GPT-2-scale models used in much of the related work, our 8B-A1B MoE setting is closer to realistic LLM training.
For additional evidence, we have added FA training experiments on a 10B-A1.6B model, with results included in Appendix F.4 of the revision. The router study also includes analyses of other MoE models. Table 1 and Fig. 5 of our paper compare weight similarity and actual routing distributions across models to examine whether parameter similarity consistently characterizes routing behavior.
Sections 4.1–4.2 and Appendix C of our paper already specify the main training configuration, fault-injection procedure, and monitoring settings. Rebuttal Table 1 summarizes these details. In particular, Eqs. (14)–(16) in Appendix C provide the bit-level operations, a concrete example, and the error expansion. We will consolidate this information in the revision for easier reference.
Rebuttal Table 1: Setup of the main FA experiments