- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a new training method for MLM-based models. Evaluations on multiple classical tasks (e.g., GLUE) and languages (bn, en, ta) are provided.
Critical strengths
Interesting, albeit outdated, problem. Thorough experiments covering classification and generation. I particularly appreciate the focus of this paper on languages other than English.
Critical weaknesses
I think this paper has good potential, but needs some significant proofreading and improvement on experimental protocols.
Here's how I would improve this paper: Writing:
- You need to proofread your work. A few examples: the citations macros are all wrong. You reintroduce MLM's abbreviation multiple times.
- The conclusion should tie to your thesis, not just say 'this confirms that' and point to the appendix for the sections that are missing.
- Models are lacking citations.
- No discussion in the main body of the paper of the limitations of the work (if only a few) do not provide trust.
Technically:
- The maths are lacking. What is ? What is ? ?
- You need to add proof sketches.
- It is not clear to me if the models were finetuned or trained from scratch--this affects the full suite of results, including the use of GLUE (re my question below).
- No stat sig tests. Achieving +X% in some metric can or cannot be significant depending on a large amount of variables. This is a must. Standard deviations are in the appendix, but I'd not consider these stat sig and should be in the main body.
Questions for authors
- It is not clear to me why is Hindi in the appendix.
- Why is MLM still relevant over autoregressive mechanisms?
- Did you train the models from scratch or finetuned them (it is unclear what 'backbone' is)?