- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper reports on hyper-parameter (HP) search for supervised NER. Common hand-engineered learning rate schedules are compared with BO- and RL-based methods, finding a remarkable effectiveness of simple baselines like Random Search.
Critical strengths
The paper is - to a large extent- a negative result paper, and should be encouraged as such.
The experiments are described carefully and clearly.
Critical weaknesses
The paper was not an exciting read to me as the novelty is limited. Still, a straightforward exploration of existing techniques on existing benchmarks like this can be relevant to practioners. However, the selected search budget is so small that - as the authors acknowledge - some of the methods (like step decay and reducelronplateau) collapse to a constant LR, and others (like RL or hyperband) are simply not suitable. So the set of evaluated methods does not seem to be well aligned with the experimental setting, and that limits the reliability of the results.
Presentation-wise there is room for improvement. I think there are better ways to visualize the results than in a single dense results table. This is particularly true in this case as the numbers in Table 1 are pretty much all over the place (which is admittedly part of the authors’ point). Some more insights into why method/task combinations were better than others (how did the optimized HPs differ) would be helpful.
Questions for authors
-
Did you see patterns in the low- and high-performing optimized HPs?
-
How do you think would a different search budget or different parameters for the optimization methods change the results?
We re-ran the entire selection stage at 10-epoch fidelity — it changes the conclusion
Our own fidelity measurement said the problem was not the optimiser but the trial length: at 2 epochs the validation score was anti-predictive of the final result (ρ = −0.586), and only at 10 epochs did it become faithful (ρ = +1.00). So we re-ran the whole search stage with 10-epoch trials, all five selectors, both backbones, 3 seeds, then retrained each selected configuration for the common 10 epochs.
Failure modes disappear
Across both backbones and both datasets, 60 cells produced no degenerate selection and no collapsed retrain. This is the most robust effect we have found, and it is the direct consequence of the fidelity curve.
Results (final-epoch test F1, 3 seeds)
Per cell: 14 of 20 selector × dataset × backbone cells now beat the static baseline (RoBERTa 9/10, BERT 5/10). All ten Wikigold cells beat static, for both backbones, by +0.016 to +0.061. On Webpage the gain appears on RoBERTa (four of five selectors, up to +0.077 for Random Search) and does not transfer to BERT.
The honest reading
random_trials: 10), so it is one result, not two.What we now claim: on distantly-supervised NER the binding constraint on configuration selection is the fidelity of the trial, not the choice of optimiser. At 2 epochs the criterion is inverted and selection is worse than a static baseline; at 10 epochs the criterion is faithful, degenerate selections vanish, and selection beats the baseline on Wikigold for both backbones and on Webpage for RoBERTa. The remaining Webpage/BERT interaction we do not explain, and we will report it as an open failure rather than average it away.