- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents EMPIRE, an integer linear programming formulation for partitioning NER datasets. It balances entity overlap and contextual similarity through a weighting parameter while constraining split sizes and entity-class distributions. Experiments on 24 datasets, with downstream evaluation using BiLSTM-CRF and Llama-2-7B, show that reducing entity overlap lowers test F1 and that class-balance constraints help avoid confounding this reduction with insufficient training support.
Critical strengths
none
Critical weaknesses
- Limited conceptual novelty. Entity leakage and memorization in NER are well studied. The main findings largely confirm prior work, while the conceptual advance from jointly controlling context leakage and class balance remains limited.
- Limited methodological novelty. EMPIRE combines established techniques within an integer programming formulation. Because the baselines lack equivalent class-balance and split-size constraints, the experiments do not clearly isolate the benefit of the proposed formulation.
Questions for authors
- How does EMPIRE compare with an entity-grouping or graph-partitioning baseline under equivalent split-size and class-balance constraints? This comparison would help isolate the contribution of the proposed optimization formulation.
- How much of the difference between α = 1 and α = 0.5 is attributable to objective weighting rather than assignment granularity, given that the former assigns individual sentences and the latter assigns clusters?
3 comments
Response to Reviewer fYPh (continued)
Question-1:
MinCut is our graph-partitioning baseline and already uses the same split-size targets as EMPIRE, without class-balance constraints. Setting in EMPIRE drops our class-balance constraint too, matching the two methods. As shown in Weakness-2, at matched class drift (0.1407 vs. MinCut's 0.1403), EMPIRE () reaches 96.71% entity disjointness vs. MinCut's 95.19%, while matching native split ratios on 24 of 24 datasets vs. MinCut's 23 of 24.
Question-2:
Our experimental design does not separate the objective weight from granularity between and , and we now state this plainly in Section 5.2. Three things change together with : (1) the objective weight, (2) the K-means clustering space in Appendix A.3, so cluster membership changes with , and (3) assignment granularity at ( individual sentences vs. clusters for ). Section 5.2 previously noted (3) and has been updated to state (2) as well.
Granularity is held fixed when comparing against , since both cluster at . At and , moving from to raises entity disjointness by 2.74 points and lowers ECR by 7.99, while information leakage rises by 0.0127 and test F1 drops by 0.025, matching the direction expected from the weight change. Even in that pair, the K-means clusters shift with , and comparing with also changes granularity. So we do not isolate the weight on its own, and we have clarified in Section 5.2 and Appendix A.11 that our results describe the three complete configurations we ran.
Action: We have stated in Section 5.2 and Appendix A.3 that enters both the K-means clustering features and the objective, presented vs. as our fixed-granularity comparison, and described as a middle configuration between the two extremes.
Response to Authors
I appreciate the authors’ work during the rebuttal period. Their response addresses some of my questions, though I still maintain my concerns regarding the paper’s conceptual and methodological novelty. Given these clarifications, I will raise my score to 2.
Response to Reviewer fYPh
We thank the reviewer for the feedback and have uploaded a revised PDF (changes in red) making our contributions and matched baseline comparisons clearer.
Weakness-1:
Our splitting formulation contributes three specific pieces:
1. Controlling entity and context leakage in one objective. MinCut targets entity overlap alone, while similarity-based splitters target context alone. No prior splitter combines both or exposes their trade-off through a weight α. Having both in one program under identical constraints is what lets us sweep α between the two extremes to test how each type of overlap affects NER scores.
2. A mention-level class-balance constraint for NER. Existing stratified splitters, including DataSAIL, bound how many sentences contain each class. In NER, training support depends on how many mentions of each class reach a split, which requires weights indexed by
(element, class)instead ofelementalone (see also Weakness-2 below).3. Showing how missing class support distorts leakage-reduced evaluations. Prior work removes entity overlap and reads the resulting F1 drop as a generalization gap. Section 5.4 shows that a large part of that drop is not about unseen entities at all: removing overlap without class-balance bounds leaves 20–32 classes completely missing from a split at δ=1 (vs. 5–6 at δ∗), and leaves 2–8 datasets with fewer than 200 gold test entities. Figure 3b shows that once training mention counts are matched, unseen test entities are recovered at the same rate under both δ=1 and δ∗. What changes is how many test entities belong to classes with almost no training examples: 5.8% fall in classes with below ten training mentions at δ=1, vs. 0.1% at δ∗. Because of that, every δ=1 setting drops F1 by 0.13–0.20 below Native, compared with 0.009–0.087 at δ∗. This changes how existing leakage-reduced results should be interpreted as well as how future splits should be built.
We fully agree that entity leakage and memorization in NER are already well established in prior work showing higher scores on seen vs. unseen entities (Augenstein et al., 2017; Fu et al., 2020; Elangovan et al., 2021; Golde et al., 2025; Babl et al., 2025). We do not claim to discover that effect; our contributions are the joint mention-balanced formulation and showing how low class support distorts unconstrained evaluations.
Action: We have rewritten the contributions in Section 1 to separate what prior work already established on entity memorization (citing the papers above) from what our formulation and class-support analysis add.
Weakness-2:
The constraint-matched comparison with MinCut is already in our tables, and we should have highlighted it directly.
MinCut uses the same split-size targets as EMPIRE without enforcing class balance. Setting δ=1 removes EMPIRE's class-balance constraint as well, matching MinCut on both split-size and class-balance rules:
At matched class drift (0.1407 vs. 0.1403, a difference of 0.0004), EMPIRE reaches 1.52 points higher entity disjointness than MinCut while keeping native split ratios on all 24 datasets, whereas MinCut changes AnatEM's training share (0.50→0.51) to reach 100% disjointness (Section 5).
This separates the two effects directly: comparing MinCut with EMPIRE (α=1,δ=1) measures the formulation's gain at matched drift, while moving from δ=1→δ∗ measures the class constraint's effect (cutting drift from 0.1407 to 0.0179, below Native's 0.0415, for 2.09 points of disjointness). For DataSAIL, a matched comparison is not possible because its stratification weight has no class index and cannot bound mention counts per class.
Action: We have restructured Section 5.1 to compare EMPIRE (α=1,δ=1) directly against MinCut under matched constraints, and presented the δ=1→δ∗ step separately as the effect of class balancing.