acceptodds
Under review as a conference paper at ICLR 2027

SANLAPA: Adversarial Masked Language Modeling via Differentiable Teacher-Student Minimax Pre-training

Abstract

Masked Language Modeling (MLM) has been central to the success of bidirectional encoders, but its standard stochastic masking strategy often allocates training signal to easily predictable tokens, limiting pre-training efficiency. We present SANLAPA (which means "dialogue"), an adversarial pre-training framework that replaces random masking with an adaptive masking policy learned through a teacher-student minimax game. A lightweight Teacher identifies tokens or spans that maximize the Student's reconstruction loss, while the Student learns to recover them. To enable end-to-end optimization, we introduce a differentiable masking layer based on Gumbel-Softmax relaxation together with a sequential joint training schedule that stabilizes the adversarial dynamics. Unlike prior adversarial ELECTRA-style RTD or AMOS approaches, SANLAPA retains reconstruction as the core learning objective and adversarially optimizes what to mask. We validate the framework across encoder-only and encoder-decoder architectures, including ALBERT, ModernBERT, ELECTRA, DeBERTaV3, T5, and ByT5, while additionally comparing against the adversarial AMOS baseline on Bangla, English, and Tamil. Across 10 standard English benchmarks and 22 Indic downstream dataset settings, SANLAPA yields consistent improvements over corpus- and step-matched baselines, with peak gains of up to +3.13 percentage points (pp) Composite score on discriminative tasks and +4.58 pp Composite score on generative tasks, without increasing the parameter count of the final deployed model. These results show that learning what to hide provides a strong and scalable alternative to fixed stochastic masking.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
adversarial training, pre-training, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Y8GGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new training method for MLM-based models. Evaluations on multiple classical tasks (e.g., GLUE) and languages (bn, en, ta) are provided.

Critical strengths

Interesting, albeit outdated, problem. Thorough experiments covering classification and generation. I particularly appreciate the focus of this paper on languages other than English.

Critical weaknesses

I think this paper has good potential, but needs some significant proofreading and improvement on experimental protocols.

Here's how I would improve this paper: Writing:

  1. You need to proofread your work. A few examples: the citations macros are all wrong. You reintroduce MLM's abbreviation multiple times.
  2. The conclusion should tie to your thesis, not just say 'this confirms that' and point to the appendix for the sections that are missing.
  3. Models are lacking citations.
  4. No discussion in the main body of the paper of the limitations of the work (if only a few) do not provide trust.

Technically:

  1. The maths are lacking. What is θT\theta_T? What is MsoftM_{soft}? E\mathbf{E}?
  2. You need to add proof sketches.
  3. It is not clear to me if the models were finetuned or trained from scratch--this affects the full suite of results, including the use of GLUE (re my question below).
  4. No stat sig tests. Achieving +X% in some metric can or cannot be significant depending on a large amount of variables. This is a must. Standard deviations are in the appendix, but I'd not consider these stat sig and should be in the main body.
Questions for authors
  1. It is not clear to me why is Hindi in the appendix.
  2. Why is MLM still relevant over autoregressive mechanisms?
  3. Did you train the models from scratch or finetuned them (it is unclear what 'backbone' is)?
Reviewer_TZTVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SANLAPA, an adversarial masked language modeling framework that learns an adaptive masking strategy through teacher–student training. The teacher selects tokens or spans that are difficult for the student to reconstruct, while the student learns to recover the masked content. The method uses differentiable masking and sequential joint optimization to stabilize training. Extensive experiments across multiple languages, model architectures, and downstream tasks demonstrate consistent improvements over existing baselines.

Critical strengths

S1. The paper addresses a meaningful limitation of random masking in language model pre-training, and the proposed adaptive masking strategy is intuitive and well motivated.

S2. The teacher–student minimax framework is straightforward, and the differentiable masking mechanism and sequential optimization strategy form a coherent solution.

S3. This work provides evaluations across multiple languages, model architectures, and downstream tasks, with detailed ablation studies supporting the effectiveness of the proposed approach.

Critical weaknesses

W1. I think the relationship between token difficulty and learning value could be further discussed. Some tokens may be inherently difficult to predict from the context, but masking them may not necessarily contribute to better representation learning. The authors also acknowledge this issue in Sections 6.1 and 6.2, where the teacher may tend to select OOV tokens or rare entities. I think a more detailed discussion of the relationship between token difficulty and actual learning value would help better justify the proposed approach.

W2. An additional baseline could help clarify the role of the proposed mechanism. I think it would be useful to include an additional baseline such as that directly uses the student's uncertainty to adjust masking probabilities, without introducing a separate teacher. Such a comparison could help clarify whether the improvements mainly come from dynamically selecting difficult tokens or from the teacher–student minimax framework itself.

Questions for authors

My questions are related to the weaknesses mentioned above.

Reviewer_FtU9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces an adversarial masking policy in masked language model pretraining. The adversarial masking policy introduces a teacher to maximize training loss through token selection, and a follow-up student to minimize the overall training loss given the tokens selected by the teacher. To further stabilize training, the authors introduce Gumbel-Softmax with a straight-through estimator and sequential updates, along with entropy regularization. The experiment section evaluates this method in a from-scratch pretraining scenario across several languages, covering both encoder-only and encoder-decoder architectures.

Critical strengths
  1. The evaluation across languages and architectures is broad, extending previous work to pre-training from scratch scenarios. These scenarios provide useful evidence about the applicability of learning masking during training.

  2. The reported downstream gains and FLOPs to target improvements suggest the potential practical benefits of the method.

Critical weaknesses
  1. Incompatible positioning. This paper omits a very close prior method [1] which already combines adversarial mask selection and an MLM reconstruction objective, along with Gumbel-Softmax-based differentiation. The broader evaluation is valuable, but the advances over this closely correlated framework still need the authors' sufficient analysis.

  2. Unspecified differentiable masking. The paper does not clearly describe the settings of the Gumbel-Softmax temperature or its schedule (or potential schedule) during training. Also, if I am correct, the definition of the continuous top-k masking during the backward pass is not specified in the paper. These details are important, and as far as I know, the Gumbel-Softmax with straight-through estimator is very sensitive and might introduce instability during training. Therefore, introducing these parameters as well as their settings is important for reproducibility.

  3. The paper claims that a stronger teacher might not help the student learn better, but whether the teacher is actually stronger is not well established. As the teacher model's capacity gets larger, it becomes harder to converge during training, and the capability of the teacher to maximize the student's loss is not well studied in the paper.

I think it would be better if the authors could provide quantitative results on the loss maximization ability of teacher models with different capacities. For example:

  1. Averaging the loss over the trajectory, or

  2. Using a fixed candidate student to test the loss maximization capability of different models at different points along the trajectory.

Moreover, entropy ablation reports focus on downstream scores, but a study directly comparing the overall coverage of masked tokens would help readers better establish an intuitive understanding of how different models change that coverage.

[1] Vu, Thuy, Dinh Phung, and Gholamreza Haffari. "Effective unsupervised domain adaptation with adversarially trained language models." In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 6163-6173. 2020.

Questions for authors
  1. What is the substance that distinguishes the proposed method from Vu et al [1].?

[1] Vu, Thuy, Dinh Phung, and Gholamreza Haffari. "Effective unsupervised domain adaptation with adversarially trained language models." In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp. 6163-6173. 2020.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.