acceptodds
Under review as a conference paper at ICLR 2027

Automated Hyperparameter Configuration Selection for Distantly Supervised NER using Reinforcement Learning and Bayesian Optimization

Abstract

Distantly supervised Named Entity Recognition (NER) produces noisy, incomplete labels that make training highly sensitive to hyperparameter choices. We study automated hyperparameter configuration selection for this setting, comparing Bayesian Optimization (Gaussian Process Thompson Sampling, GP-TS) and reinforcement learning (Twin Delayed DDPG, TD3; Soft Actor-Critic, SAC) against resource-aware hyperparameter optimization baselines (Random Search, Tree-structured Parzen Estimator, TPE) and hand-designed learning-rate schedules (Step Decay, Cosine Annealing, ReduceLROnPlateau). Each method selects a configuration using short-budget 2-epoch trials, after which the selected configuration is retrained for 10 epochs, so that every method is compared under an identical final training budget. Using BERT-CRF and RoBERTa-CRF under the BOND distant supervision framework, we evaluate on three benchmarks (Webpage, Wikigold, CoNLL-2003) with three random seeds and report meanstandard deviation. We find that configuration selection improves over a static baseline on every dataset: on CoNLL-2003, Random Search reaches 0.719 (RoBERTa-CRF) and 0.737 (BERT-CRF) against static baselines of 0.674 and 0.681, with GP-TS, TPE, TD3, and SAC all within 0.02; on Wikigold, TD3 and Random Search reach 0.480 and 0.481 against 0.452; on Webpage the gains are smaller but positive for the strongest selectors (SAC 0.558, Random Search 0.557 vs. 0.524). The hand-designed schedules do not help at this budget: Step Decay and ReduceLROnPlateau fall below the static baseline on Wikigold and CoNLL-2003, because neither reaches its first decay step within 10 epochs. We further show that search-stage scores from 2-epoch trials are unreliable proxies for 10-epoch retraining performance, and that best-checkpoint scores — the usual reporting convention — are indistinguishable across schedules here because the best checkpoint almost always falls in the first epoch or two, which is why we report final-epoch scores. Our results show that automated configuration selection is a viable, complementary strategy under weak supervision, and they provide practical guidance for evaluating such methods fairly.

Primary area:
reinforcement learning
Keywords:
distantly supervised NER, hyperparameter optimization, Bayesian optimization, reinforcement learning, Thompson sampling, weak supervision, configuration selection, learning rate scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LaYXOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper reports on hyper-parameter (HP) search for supervised NER. Common hand-engineered learning rate schedules are compared with BO- and RL-based methods, finding a remarkable effectiveness of simple baselines like Random Search.

Critical strengths

The paper is - to a large extent- a negative result paper, and should be encouraged as such.

The experiments are described carefully and clearly.

Critical weaknesses

The paper was not an exciting read to me as the novelty is limited. Still, a straightforward exploration of existing techniques on existing benchmarks like this can be relevant to practioners. However, the selected search budget is so small that - as the authors acknowledge - some of the methods (like step decay and reducelronplateau) collapse to a constant LR, and others (like RL or hyperband) are simply not suitable. So the set of evaluated methods does not seem to be well aligned with the experimental setting, and that limits the reliability of the results.

Presentation-wise there is room for improvement. I think there are better ways to visualize the results than in a single dense results table. This is particularly true in this case as the numbers in Table 1 are pretty much all over the place (which is admittedly part of the authors’ point). Some more insights into why method/task combinations were better than others (how did the optimized HPs differ) would be helpful.

Questions for authors
  • Did you see patterns in the low- and high-performing optimized HPs?

  • How do you think would a different search budget or different parameters for the optimization methods change the results?

Reviewer_RUfmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the automated hyperparameter configuration system for Distantly Supervised NER task, comprising diverse hyperparameter optimization methods: GP-TS (Bayesian Optimiziation), TD3 (RL), SAC (RL), Random Search, TPE. Experiment results shows that hyperparameter configuration selection itself helps to enhance NER performance without the modification of model's architecture or label quality itself, compared to using hand-set configuration.

Critical strengths
  • This paper propose a new hyperparameter configuration system for Distantly Supervised NER task. While prior works mostly focus on building an architecture or label itself, this paper focuses on its hyperparameter configuration to enhance its performance. Such approach seems to be first and novel at this Distantly Supervised NER task, since it can be co-used complemantary to any NER competing methods to enhance its performance.
  • Proposed automated hyperparameter configuration system for Distantly Supervised NER task contains diverse type of hyperparameter optimization methods including GP-TS (Bayesian Optimiziation), TD3 (RL), SAC (RL), Random Search, TPE, and diverse experiments has compared each methods at diverse benchmarks, and it additionally showed proposed system's robustness by adapting it to competitive baselines: RoBERTa-CRF, BERT-CRF, and BOND.
Critical weaknesses
  • Limited methodological novelty and interpretation. Since this paper applies existing HPO methods to NER tasks without introducing additional methodological insights, its contribution depends on the depth of its analysis and interpretations of the results. I assume the current analysis seems to be a bit limited, since it does not provide sufficient interpretation for the observed result. For example, random search has mostly shown good performance for NER tasks, but the interpretation for search space such as hyperparameter search landscape are sufficiently smooth or or have multi-mode, allowing random sampling to easily discover high-performing configurations seems lacked.
  • Questionable evaluation protocol. Even though 2 epoch training are used for hyperparameter selection, the final-epoch performance after 10 epoch training is reported. But the authors have written that best checkpoints typically occur at first 1-2 epochs. If 2 epoch was sufficient, it seems like authors didn't have to enlarge the epoch to 10, and use 2 epoch training for both hyperparameter selection and performance report. But as authors said, if the performance report was indistinuguishible for 2 epoch, the authors should have use 10 epoch training and final-epoch for both hyperparameter selection and performance report.
Questions for authors
  • Please provide the iteration vs performance result of each hyperparameter optimization methods, and interpretation of why the variance is large at webpage, while it shows relatively small at wikigold may further resolve my question.
  • And no tendency seems shown at the searched hyperparameters. Is there some multi-mode landscape? why hand-designed scehdule underperformed in wikigold? what was the difference between hand-design hyperparam and discovered one?
Reviewer_C1eYOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper compares Bayesian optimization, reinforcement learning, random search, and TPE for selecting hyperparameters in distantly supervised NER. Configurations are selected using short training runs and then retrained for ten epochs. Across three datasets and two backbones, tuning often improves over the baseline, but the more complex methods do not consistently outperform random search.

Critical strengths

The inclusion of random search and TPE is useful. Showing that simpler methods can match more expensive selectors is practically useful, although the weaknesses described below make that conclusion difficult to assess confidently.

Critical weaknesses
  • Several basic experimental details contradict each other. The baseline learning rate is 5x10^-5 in Section 4.1 but 1x10^-5 in the appendix, which changes the interpretation of the selected learning rates. Figure 2 shows a best F1 of 0.0293, while its caption says 0.095. Table 1 claims three seeds throughout, but Section 4.2 describes one column as single-seed despite reporting standard deviations. These discrepancies make it unclear which experimental setup produced the results.
  • The comparisons do not support the paper’s conclusions about tuning methods and schedules. SAC receives 50 search trials while GP-TS and TD3 receive 20, so the search budgets are not matched. The schedule comparisons are also not convincing. Neither Step Decay nor ReduceLROnPlateau changes the learning rate during training, so the results don't prove that scheduling does not help.
  • The authors also leave out results from the checkpoints that performed best on the validation set, which they say were similar across schedules, and report only scores at the end of training.
  • The appendix seems to describe DQN, TRPO, and GRPO without testing them?
Questions for authors
  • What learning rate, number of search trials, and number of seeds were used for Table 1? Why do these differ across sections, and why does Figure 2 disagree with its caption?

  • Do the gains remain when methods get the same search budget, schedules actually change the learning rate, and checkpoints are chosen by validation performance?

Comments on the paper

2 comments

We re-ran the entire selection stage at 10-epoch fidelity — it changes the conclusion

Our own fidelity measurement said the problem was not the optimiser but the trial length: at 2 epochs the validation score was anti-predictive of the final result (ρ = −0.586), and only at 10 epochs did it become faithful (ρ = +1.00). So we re-ran the whole search stage with 10-epoch trials, all five selectors, both backbones, 3 seeds, then retrained each selected configuration for the common 10 epochs.

Failure modes disappear

2-epoch protocol10-epoch protocol
degenerate selections (score < 0.05)26.7 % (4/15) on Webpage0 / 60
retrains that collapsed to F1 = 022 % on Webpage0 / 60

Across both backbones and both datasets, 60 cells produced no degenerate selection and no collapsed retrain. This is the most robust effect we have found, and it is the direct consequence of the fidelity curve.

Results (final-epoch test F1, 3 seeds)

pooledstaticΔ
Webpage / RoBERTa0.56320.524+0.0392
Wikigold / RoBERTa0.50580.452+0.0538
Wikigold / BERT0.47470.456+0.0187
Webpage / BERT0.50530.548−0.0427

Per cell: 14 of 20 selector × dataset × backbone cells now beat the static baseline (RoBERTa 9/10, BERT 5/10). All ten Wikigold cells beat static, for both backbones, by +0.016 to +0.061. On Webpage the gain appears on RoBERTa (four of five selectors, up to +0.077 for Random Search) and does not transfer to BERT.

The honest reading

  • The fidelity fix is real and replicates. It removes degenerate selections and collapsed retrains on both backbones, and it turns selection clearly positive on Wikigold for both.
  • It is not a universal win. Webpage/BERT stays below the static baseline (−0.043), so "10 epochs fixes everything" would be wrong.
  • It is not compute-matched. 10-epoch search uses 60 epoch-units against 40 for the 2-epoch protocol, a 1.5× increase. This is a statement that fidelity matters, not that the method is better per unit compute.
  • It is still not the optimiser. At 10 epochs Random Search and TPE are the strongest on Webpage (0.5988 / 0.6005) and GP-TS is the only RoBERTa selector below the baseline. TPE at six trials is pure random sampling (random_trials: 10), so it is one result, not two.

What we now claim: on distantly-supervised NER the binding constraint on configuration selection is the fidelity of the trial, not the choice of optimiser. At 2 epochs the criterion is inverted and selection is worse than a static baseline; at 10 epochs the criterion is faithful, degenerate selections vanish, and selection beats the baseline on Wikigold for both backbones and on Webpage for RoBERTa. The remaining Webpage/BERT interaction we do not explain, and we will report it as an open failure rather than average it away.

Four follow-up questions, answered with measurements

We ran targeted diagnostics rather than speculating. Everything below is 10-epoch fidelity on Webpage, 9 sampled configurations per setting plus BOND's default configuration, seed 42 unless stated.

1. Would a better data split help? — It improves the signal, but does not change the outcome

We built a gold dev split (67 of the 135 human-annotated test documents as dev, the other 68 as test) and repeated the identical configurations against it.

dev criterionbackboneρ(dev, test)mean gap (test − dev)
distant dev (current)RoBERTa+0.818+0.185
distant dev (current)BERT+0.886+0.145
gold devRoBERTa+0.982−0.002
gold devBERT+0.941+0.004

A gold dev is clearly the better criterion: the ranking becomes near-perfect and the 0.15–0.19 F1 scale bias disappears. That is the quantitative case for the point Reviewer LaYX and RUfm both made about validation quality.

But the selected configuration does not change. Taking the dev-best config and scoring it on the held-out gold test:

dev criterionRoBERTa (static 0.524)BERT (static 0.548)
distant dev0.5847 (+0.061)0.4685 (−0.080)
gold dev0.5906 (+0.067)0.4681 (−0.080)

Gold dev buys +0.006 on RoBERTa and nothing at all on BERT. The ranking was already good enough; the binding constraint is elsewhere.

2. Why does Webpage/BERT regress? — Nothing we sample is as good as the default

best test F1 among 9 sampled configsBOND default (3 seeds)
RoBERTa0.58470.5497
BERT0.47140.5457

On BERT the sampled search space lies entirely below BOND's default configuration. Selection cannot return a configuration better than the best one available to it, so the regression is a search-space failure, not a selection failure. This is the direct answer to the Webpage/BERT question, and it is not something a better split or a better optimiser can fix.

3. Why does random search beat the adaptive optimisers?

Across 48 cells with trial-level records, we compared each adaptive method's guided phase against its own random warm-up:

  • the guided phase raises the dev score by +0.0275 on average (34/48 cells improve);
  • GP-TS specifically raises it by +0.051 (9/12 cells).

So the optimisers work — on the surrogate. But:

Spearman ρ(adaptive dev gain, retrained test F1) = +0.063, p = 0.67.

Improving the validation score is statistically unrelated to improving the final model. GP-TS produced the largest dev gain and the lowest mean test score of the four methods (0.509 vs random search 0.524). An optimiser that can only see the surrogate is therefore guaranteed to be no better than random search, and over-committing to the surrogate makes it worse.

4. What would make selection more effective

Put the default configuration in the candidate pool. Its dev score is the highest of anything we measured — 0.2616 (RoBERTa) and 0.2559 (BERT) over three seeds, against 0.2373 and 0.2386 for what the search actually selects — so dev-selection would often choose it.

search, 15 cellsdefault configwith default as a candidate
BERT0.50530.5457+0.040
RoBERTa0.56320.5497−0.013

A net benefit, and it makes the worst case equal to the baseline. It is not free — on RoBERTa it forgoes a small gain — but one extra trial is a cheap bound on the downside, and no new labels are needed. Two supporting changes follow from the same evidence: use a margin rule rather than taking the argmax (since ρ ≈ 0), and note that investment in a better optimiser cannot help while surrogate gains do not transfer.

Caveats. 9 configurations per setting, one seed for the probes except the default (3 seeds); the gold split halves the test set, so each half is 67–68 documents and differences of 1–2 F1 are not resolvable. The sampled space was log-uniform over lr/weight decay/betas/warmup/batch; a default-centred space would behave differently, which is precisely the recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.