SANLAPA: Adversarial Masked Language Modeling via Differentiable Teacher-Student Minimax Pre-training

Masked Language Modeling (MLM) has been central to the success of bidirectional encoders, but its standard stochastic masking strategy often allocates training signal to easily predictable tokens, limiting pre-training efficiency. We present SANLAPA (which means "dialogue"), an adversarial pre-training framework that replaces random masking with an adaptive masking policy learned through a teacher-student minimax game. A lightweight Teacher identifies tokens or spans that maximize the Student's reconstruction loss, while the Student learns to recover them. To enable end-to-end optimization, we introduce a differentiable masking layer based on Gumbel-Softmax relaxation together with a sequential joint training schedule that stabilizes the adversarial dynamics. Unlike prior adversarial ELECTRA-style RTD or AMOS approaches, SANLAPA retains reconstruction as the core learning objective and adversarially optimizes what to mask. We validate the framework across encoder-only and encoder-decoder architectures, including ALBERT, ModernBERT, ELECTRA, DeBERTaV3, T5, and ByT5, while additionally comparing against the adversarial AMOS baseline on Bangla, English, and Tamil. Across 10 standard English benchmarks and 22 Indic downstream dataset settings, SANLAPA yields consistent improvements over corpus- and step-matched baselines, with peak gains of up to +3.13 percentage points (pp) Composite score on discriminative tasks and +4.58 pp Composite score on generative tasks, without increasing the parameter count of the final deployed model. These results show that learning what to hide provides a strong and scalable alternative to fixed stochastic masking.