Full-Stack NVFP4: Stable, Role-Aware 4-Bit LLM Pretraining Beyond Linear Projections

NVFP4 pretraining typically targets Transformer projections while leaving optimizer states, optimizer computation, and attention in higher precision. Extending NVFP4 beyond projections is not a uniform quantization problem: each module has a distinct numerical role and failure mode. We present $\text{Full-Stack NVFP4}$, a role-aware framework instantiated by four independently composable recipes. $\text{Subspace-Protected LoRA-SVD}$ retains a compact BF16 projection subspace within full-shape NVFP4 computation; gradient decoupling and periodic SVD realignment stabilize its low-rank factors, reducing the linear-only loss gap from $\text{1.40\%}$ to $\text{0.61\%}$. $\text{NVFP4-State AdamW}$ transforms persistent momentum states before storage, $\text{NVFP4 Root}$ stabilizes direct NVFP4 Newton–Schulz iterations, and $\text{Selective FP4 Attention}$ protects softmax-sensitive products in BF16. On 3B pretraining with 64B tokens, BF16 and Full-Stack NVFP4 reach losses of $\text{2.267}$ and $\text{2.286}$, a $\text{0.838\%}$ gap, with similar aggregate zero-shot results. Native four-block measurements on one RTX 5090 show 2.50–2.83$\times$ Root speedups over optimized BF16 and 37.9–42.5% lower AdamW peak memory.