acceptodds
Under review as a conference paper at ICLR 2027

EMPIRE: Explicit Method for dataset Partitioning using Integer programming for Named Entity Recognition to reduce data leakage

Abstract

Evaluations of Named Entity Recognition (NER) models are often confounded by data leakage between dataset splits. Entity leakage occurs when the same entity appears in both training and test data, and context leakage when test sentences share their surrounding context with training sentences; both let models score well through memorization rather than generalization. We introduce EMPIRE (Explicit Method for dataset Partitioning using Integer programming for Entity Recognition), which formulates dataset splitting as an integer linear program. It reduces entity and context leakage, weighted by a single parameter , while fixing the split sizes and bounding each entity class's share of every split. On 24 NER datasets, EMPIRE matches the entity disjointness of a minimum-cut baseline while preserving class balance, and with it reduces context leakage more than a similarity-based baseline. Removing entity leakage lowers the test F1 of both a BiLSTM-CRF and a fine-tuned Llama-2-7B, whereas removing context leakage alone does not, indicating that much of the F1 on native splits reflects entities seen during training. The class-balance constraint is essential for this comparison: without it, splits strand entity classes and lower F1 for reasons unrelated to generalization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Data leakage, NER, Dataset Partitioning, Integer Linear Programming, Entity Leakage, Context Leakage

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fYPhOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents EMPIRE, an integer linear programming formulation for partitioning NER datasets. It balances entity overlap and contextual similarity through a weighting parameter while constraining split sizes and entity-class distributions. Experiments on 24 datasets, with downstream evaluation using BiLSTM-CRF and Llama-2-7B, show that reducing entity overlap lowers test F1 and that class-balance constraints help avoid confounding this reduction with insufficient training support.

Critical strengths

none

Critical weaknesses
  1. Limited conceptual novelty. Entity leakage and memorization in NER are well studied. The main findings largely confirm prior work, while the conceptual advance from jointly controlling context leakage and class balance remains limited.
  2. Limited methodological novelty. EMPIRE combines established techniques within an integer programming formulation. Because the baselines lack equivalent class-balance and split-size constraints, the experiments do not clearly isolate the benefit of the proposed formulation.
Questions for authors
  1. How does EMPIRE compare with an entity-grouping or graph-partitioning baseline under equivalent split-size and class-balance constraints? This comparison would help isolate the contribution of the proposed optimization formulation.
  2. How much of the difference between α = 1 and α = 0.5 is attributable to objective weighting rather than assignment granularity, given that the former assigns individual sentences and the latter assigns clusters?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fYPh

We thank the reviewer for the feedback and have uploaded a revised PDF (changes in red) making our contributions and matched baseline comparisons clearer.

Weakness-1:

Our splitting formulation contributes three specific pieces:

1. Controlling entity and context leakage in one objective. MinCut targets entity overlap alone, while similarity-based splitters target context alone. No prior splitter combines both or exposes their trade-off through a weight α\alpha. Having both in one program under identical constraints is what lets us sweep α\alpha between the two extremes to test how each type of overlap affects NER scores.

2. A mention-level class-balance constraint for NER. Existing stratified splitters, including DataSAIL, bound how many sentences contain each class. In NER, training support depends on how many mentions of each class reach a split, which requires weights indexed by (element, class) instead of element alone (see also Weakness-2 below).

3. Showing how missing class support distorts leakage-reduced evaluations. Prior work removes entity overlap and reads the resulting F1 drop as a generalization gap. Section 5.4 shows that a large part of that drop is not about unseen entities at all: removing overlap without class-balance bounds leaves 20–32 classes completely missing from a split at δ=1\delta = 1 (vs. 5–6 at δ∗\delta^\ast), and leaves 2–8 datasets with fewer than 200 gold test entities. Figure 3b shows that once training mention counts are matched, unseen test entities are recovered at the same rate under both δ=1\delta = 1 and δ∗\delta^\ast. What changes is how many test entities belong to classes with almost no training examples: 5.8% fall in classes with below ten training mentions at δ=1\delta = 1, vs. 0.1% at δ∗\delta^\ast. Because of that, every δ=1\delta = 1 setting drops F1 by 0.13–0.20 below Native, compared with 0.009–0.087 at δ∗\delta^\ast. This changes how existing leakage-reduced results should be interpreted as well as how future splits should be built.

We fully agree that entity leakage and memorization in NER are already well established in prior work showing higher scores on seen vs. unseen entities (Augenstein et al., 2017; Fu et al., 2020; Elangovan et al., 2021; Golde et al., 2025; Babl et al., 2025). We do not claim to discover that effect; our contributions are the joint mention-balanced formulation and showing how low class support distorts unconstrained evaluations.

Action: We have rewritten the contributions in Section 1 to separate what prior work already established on entity memorization (citing the papers above) from what our formulation and class-support analysis add.

Weakness-2:

The constraint-matched comparison with MinCut is already in our tables, and we should have highlighted it directly.

MinCut uses the same split-size targets as EMPIRE without enforcing class balance. Setting δ=1\delta = 1 removes EMPIRE's class-balance constraint as well, matching MinCut on both split-size and class-balance rules:

Entity disjointness (%) ↑Class drift (Hellinger) ↓Split ratios
Native84.460.0415-
MinCut95.190.1403match Native on 23 / 24
EMPIRE (α=1\alpha = 1, δ=1\delta = 1)96.710.1407match Native on 24 / 24
EMPIRE (α=1\alpha = 1, δ∗\delta^\ast)94.620.0179match Native on 24 / 24

At matched class drift (0.1407 vs. 0.1403, a difference of 0.0004), EMPIRE reaches 1.52 points higher entity disjointness than MinCut while keeping native split ratios on all 24 datasets, whereas MinCut changes AnatEM's training share (0.50→0.510.50 \to 0.51) to reach 100% disjointness (Section 5).

This separates the two effects directly: comparing MinCut with EMPIRE (α=1,δ=1\alpha = 1, \delta = 1) measures the formulation's gain at matched drift, while moving from δ=1→δ∗\delta = 1 \to \delta^\ast measures the class constraint's effect (cutting drift from 0.1407 to 0.0179, below Native's 0.0415, for 2.09 points of disjointness). For DataSAIL, a matched comparison is not possible because its stratification weight has no class index and cannot bound mention counts per class.

Action: We have restructured Section 5.1 to compare EMPIRE (α=1,δ=1\alpha = 1, \delta = 1) directly against MinCut under matched constraints, and presented the δ=1→δ∗\delta = 1 \to \delta^\ast step separately as the effect of class balancing.

AuthorsOpenReview ↗

Response to Reviewer fYPh (continued)

Question-1:

MinCut is our graph-partitioning baseline and already uses the same split-size targets as EMPIRE, without class-balance constraints. Setting δ=1\delta = 1 in EMPIRE drops our class-balance constraint too, matching the two methods. As shown in Weakness-2, at matched class drift (0.1407 vs. MinCut's 0.1403), EMPIRE (α=1,δ=1\alpha = 1, \delta = 1) reaches 96.71% entity disjointness vs. MinCut's 95.19%, while matching native split ratios on 24 of 24 datasets vs. MinCut's 23 of 24.

Question-2:

Our experimental design does not separate the objective weight from granularity between α=1\alpha = 1 and α=0.5\alpha = 0.5, and we now state this plainly in Section 5.2. Three things change together with α\alpha: (1) the objective weight, (2) the K-means clustering space [α⋅vent∥1−α⋅vstruct][\sqrt{\alpha} \cdot v_{\text{ent}} \parallel \sqrt{1 - \alpha} \cdot v_{\text{struct}}] in Appendix A.3, so cluster membership changes with α\alpha, and (3) assignment granularity at α=1\alpha = 1 (K=0K = 0 individual sentences vs. K=50K = 50 clusters for α<1\alpha < 1). Section 5.2 previously noted (3) and has been updated to state (2) as well.

Granularity is held fixed when comparing α=0\alpha = 0 against α=0.5\alpha = 0.5, since both cluster at K=50K = 50. At ϵ=0\epsilon = 0 and δ∗\delta^\ast, moving from α=0\alpha = 0 to α=0.5\alpha = 0.5 raises entity disjointness by 2.74 points and lowers ECR by 7.99, while information leakage rises by 0.0127 and test F1 drops by 0.025, matching the direction expected from the weight change. Even in that pair, the K-means clusters shift with α\alpha, and comparing with α=1\alpha = 1 also changes granularity. So we do not isolate the weight α\alpha on its own, and we have clarified in Section 5.2 and Appendix A.11 that our results describe the three complete configurations we ran.

Action: We have stated in Section 5.2 and Appendix A.3 that α\alpha enters both the K-means clustering features and the objective, presented α=0\alpha = 0 vs. α=0.5\alpha = 0.5 as our fixed-granularity comparison, and described α=0.5\alpha = 0.5 as a middle configuration between the two extremes.

Reviewer_fYPhOpenReview ↗

Response to Authors

I appreciate the authors’ work during the rebuttal period. Their response addresses some of my questions, though I still maintain my concerns regarding the paper’s conceptual and methodological novelty. Given these clarifications, I will raise my score to 2.

Reviewer_TuZjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EMPIRE, an integer linear programming framework for constructing named entity recognition dataset splits with reduced entity and context leakage. Its objective balances frequency-weighted entity disjointness against cross-split contextual similarity, subject to constraints on split sizes and per-class mention counts. Clustering is used to reduce the computational cost of context-aware optimization. The authors evaluate the framework on 24 NER datasets against native splits, MinCut, and TopSim, using split-quality metrics and downstream BiLSTM-CRF performance, with additional fine-tuned Llama-2-7B experiments on five datasets. The reported results show entity disjointness comparable to MinCut while better preserving class distributions. Entity-focused splitting generally lowers test F1 relative to native splits, whereas context-focused splitting produces no significant average decrease in the class-balanced BiLSTM-CRF experiments. The paper also examines how removing class-balance constraints can leave entity classes with insufficient training examples or too few test mentions, highlighting the importance of controlling class support when interpreting the effects of leakage reduction on model evaluation.

Critical strengths
  1. Useful insight into the role of class support in leakage-reduced evaluation. Section 5.4 shows that larger F1 drops can partly reflect insufficient training examples for certain entity classes, rather than a more informative test of generalization. Figure 3b supports this interpretation by showing similar unseen-entity recovery rates when class-level training support is comparable. This provides a useful lesson for benchmark design: reducing overlap alone is insufficient without considering how classes are represented across splits.
  2. A flexible splitting framework supported by broad empirical evaluation. EMPIRE combines entity and context objectives with split-size and class-balance constraints. Across 24 datasets, its entity-focused, class-constrained configuration achieves disjointness comparable to MinCut while better preserving class distributions. BiLSTM-CRF experiments on all 24 datasets and fine-tuned Llama-2-7B experiments on five further connect the generated splits to model behavior. Together, these results support its practical value for constructing and studying alternative NER evaluation splits.
Critical weaknesses

1. The contribution and guarantees of the constrained splitting framework are not sufficiently established.

(a) The distinction from closely related methods is based partly on an incorrect premise. Sections 1–2 exclude DataSAIL because it supposedly assumes one label per sample. However, DataSAIL explicitly allows multiple class memberships and combines similarity-aware splitting with stratification, size constraints, clustering, and integer programming (Joeres et al., 2025, Eq. 4). This does not make it equivalent to EMPIRE’s entity-isolation and mention-count formulation, but the paper needs to establish those specific differences rather than exclude the framework on incorrect grounds. Since MinCut and TopSim do not enforce comparable class constraints, their worse class preservation also does not isolate the benefit of EMPIRE’s particular formulation from the benefit of adding such constraints.

(b) The feasibility pre-check does not provide the coordinatewise guarantee claimed in Appendix A.2. The objective

min⁡X,ϵ,δ(ϵ+∑t=1Cδt)\min_{X,\epsilon,\delta}\left(\epsilon+\sum_{t=1}^{C}\delta_t\right)

finds a joint minimum-sum solution, not independently minimal tolerances, because all terms are coupled through the same partition variables. For example, consider four equal-sized clusters, two equal-sized splits with ε fixed to zero, and class-count vectors

A=(2,0,1,1),B=(2,1,1,0).A=(2,0,1,1),\qquad B=(2,1,1,0).

Under the stated 0.1 lower bound, the following partitions require different tolerance pairs:

\quad\Rightarrow\quad (\delta_A,\delta_B)=(0.1,0.5),$$ $$\\{1,4\\}\mid\\{2,3\\} \quad\Rightarrow\quad (\delta_A,\delta_B)=(0.5,0.1).$$ Neither class’s individual minimum can be attained simultaneously with the other’s. Thus, one selected tolerance vector cannot generally be interpreted as independent feasibility thresholds for its components. Allowing ε to vary also introduces a trade-off between size relaxation and class relaxation, rather than prioritizing exact split sizes. **(c) The constraints actually enforced in the experiments are ambiguous.** Section 4.3 states that ε = 0 throughout, whereas Table 2 describes zero as a lower bound that may be relaxed and reports changed split ratios. Likewise, Section 4.3 describes using the pre-check output δ∗, while Appendix A.2 says that experimental tolerances need not equal that output and that the pre-check is strictly diagnostic. Several tables label the constrained setting as δ = 0.1, although Table 8 includes tolerances up to 1. These distinctions matter because exact size preservation and class support are central to the claimed advantage. Moreover, Eq. (4) bounds each class’s allocation of mentions; it does not enforce the identical within-split class ratios suggested by Figure 1. **(d) The interpretation of optimization quality is stronger than the available certificates.** Table 9 includes substantial termination gaps, such as 185.7% for FabNER under α = 1 and δ = 1. These do not establish that the returned partitions are poor, but they leave their proximity to the optimum unresolved. The connected-component argument correctly gives a necessary condition for complete disjointness; it does not explain how much of a partially disjoint solution’s shortfall is unavoidable. Similarly, the cluster-count ablation supports K = 50 under the tested computational budget, rather than showing that finer clustering is generally unnecessary. Removing pairwise variables also does not eliminate the combinatorial nature of the remaining integer assignment problem, as Appendix A.5 suggests. The concern is not that every instance must be solved exactly, but that feasible solutions, proven optima, and structural impossibility are insufficiently distinguished. Together, these issues leave uncertainty about both the framework’s methodological advance and the guarantees underlying its empirical comparisons. **2. The evaluation does not adequately isolate the claimed effects of entity and context leakage.** **(a) Re-splitting does not isolate entity memorization.** Section 5.3 attributes the F1 decrease to recognition of entities seen during training. However, re-splitting changes the training and test examples themselves, potentially changing entity frequencies, sentence difficulty, topics, and other distributional properties. Class-count constraints address an important confound, but not all of these changes. The experiments demonstrate that the splitting procedure affects evaluation more directly than they establish how much Native performance is attributable to memorization. The intended deployment setting also matters: excluding recurring entities creates a particular generalization test, not automatically a more trustworthy test for every application. **(b) The comparison across α changes more than the objective weights.** The entity-only configuration assigns individual sentences, whereas configurations with α < 1 assign 50 clusters. Appendix A.3 additionally uses α to construct the clustering representation. Consequently, the comparisons change the objective, cluster composition, and feasible assignments simultaneously. Section 5.2 acknowledges part of this issue, but the results still do not cleanly identify the effect of the proposed interpolation weight. The comparison with TopSim introduces another difference: TopSim uses unmasked embeddings, whereas EMPIRE’s context objective uses entity-masked embeddings. **(c) The context metric and statistical result support a narrower conclusion than the paper draws.** Mean similarity over all cross-split sentence pairs can obscure a small number of highly similar training–test pairs. The authors acknowledge this when discussing TopSim, which lowers F1 without increasing entity disjointness. In addition, the optimized normalized-centroid surrogate differs from the sentence-level metric used for evaluation. Therefore, reducing this metric and observing no significant average F1 decrease does not establish that removing context leakage generally leaves performance unchanged. The current evidence supports a statement about the tested procedure and similarity measure, rather than a general contrast between entity and context memorization. **(d) Model selection and uncertainty remain relevant to the interpretation of F1 differences.** Appendix A.8 tunes BiLSTM-CRF hyperparameters on Native and transfers them to the other splits. This keeps configurations consistent but may favor Native through model selection. Training uses a single reported seed, while the statistical tests and error bars primarily characterize variation across datasets rather than within-dataset F1 variability. These limitations are particularly relevant to the small context-focused differences. The class-support analysis in Section 5.4 is valuable, but its stronger claim that reweighting “fully accounts for” the recovery gap is not accompanied by the resulting estimate, residual difference, or uncertainty. **(e) Metric and reporting inconsistencies further complicate verification.** Entity targets are stemmed during optimization but matched by exact surface form during evaluation; the objective is frequency-weighted, whereas disjointness weights unique targets equally. Accordingly, matching MinCut on disjointness does not imply matching it on ECR, where the class-constrained configuration retains more overlap. Figure 1’s information-leakage averages also disagree with Table 6: for example, Native is shown as approximately 0.175, whereas the table entries average approximately 0.396. Appendix A.4 similarly illustrates an unweighted entity objective rather than the frequency-weighted Eq. (9). These may be correctable reporting issues, but they need reconciliation before the associated quantitative claims can be assessed confidently.
Questions for authors

1. What is the precise methodological advance over existing constrained splitting frameworks? — Clarification and comparative evidence

Given DataSAIL’s support for multiple class memberships, which aspects of EMPIRE require a genuinely different formulation: per-class mention multiplicities, entity-identity isolation, or their combination with the context objective? Please distinguish these requirements from capabilities already present in similarity-aware stratified splitting. What evidence establishes the benefit of the proposed formulation under comparable size and class-support requirements? An appropriately adapted comparison, or a precise explanation of why the relevant adaptation is not feasible, would materially clarify the contribution.

2. Which constraints and feasibility guarantees were actually used? — Clarification of the existing method and experiments

(a) Was ε fixed to zero during both the pre-check and main optimization, or could it increase? Please reconcile Section 4.3 with Table 2 and provide the actual tolerances and split counts for configurations where the target ratios changed.

(b) Were the full per-class vectors returned by the pre-check used in the main experiments? Please clarify the relationship between the requested 0.1 lower bound, δ∗, the table labels, and Appendix A.2’s statement that the pre-check was only diagnostic.

(c) Is the intended guarantee a minimum-sum feasible tolerance vector rather than independently minimal tolerances? If so, does correcting this interpretation change any parameter-selection or infeasibility decisions, or only the accompanying explanation?

3. Which optimization and numerical claims are supported by the recorded results? — Clarification of existing outputs

(a) Please distinguish the pre-check’s achieved optimality status from its requested zero-gap stopping tolerance. Were all pre-check solutions proven optimal before the time limit? For the main solves with large gaps, what evidence supports attributing the remaining disjointness shortfall mainly to data structure rather than unresolved optimization?

(b) Please reconcile the information-leakage aggregates in Figure 1 with Table 6 and identify whether they use different metrics, subsets, or runs. Do the corrections affect only the overview figure, or any reported comparisons and conclusions? The toy example in Appendix A.4 should likewise be reconciled with the frequency-weighted objective.

4. What evidence supports the interpretation of context leakage? — Targeted evidence or restriction of the claim

Does EMPIRE’s context-focused splitting reduce test-to-training nearest-neighbor similarity or near-duplicate frequency, in addition to reducing average cross-split similarity? How should the TopSim result be interpreted once differences in entity masking and class support are taken into account?

A representative comparison using the same masked embeddings, together with a nearest-neighbor-based measurement, would help resolve this issue. Alternatively, would you restrict the conclusion to the absence of a significant average F1 decrease under the particular class-constrained, mean-similarity-based procedure tested here, rather than a general statement about context leakage?

5. How much of the observed F1 contrast can be attributed to leakage rather than other experimental changes? — Targeted evidence or restriction of the claim

(a) Is there evidence separating the effect of the objective weight from the change in assignment granularity and α-dependent clustering? A comparison with a fixed clustering and common constraints would address this specific ambiguity.

(b) What evidence separates entity exposure from the changed training/test composition and Native-based hyperparameter selection? A fixed-test-set or matched-example analysis, or a representative sensitivity check with comparable tuning and repeated training, would strengthen the attribution. Without such evidence, which narrower conclusion about splitting-procedure sensitivity would you retain?

These are alternative ways to support the current interpretation, not a request to repeat every configuration on all 24 datasets.

6. Can the class-support explanation be quantified explicitly? — Clarification of a central positive result

Section 5.4 states that applying the δ = 1 recovery rates to the δ∗ support distribution fully accounts for the recovery gap. Could you report the support-band weights, observed recovery rates, reweighted estimate, and remaining difference, with an indication of uncertainty? Please also clarify whether this accounting holds within datasets or primarily after pooling them. This would help determine how strongly the analysis supports class support as the explanation, rather than as one important correlate of the different splits.

13 comments
AuthorsOpenReview ↗

Response to reviewer TuZj

We thank the reviewer for the detailed and helpful review. Below we respond to each point and note the updates made in the revised PDF (in red).

Weakness-1(a) (Comparison with DataSAIL): Thank you for pointing this out. We address both parts of this comment: first how our class constraint differs formally from DataSAIL's, and next how EMPIRE compares with MinCut under matched constraints.

DataSAIL does allow multi-class labels and applies stratification to each class separately. The real difference is what the constraint counts. DataSAIL counts how many sentences contain a class, while learning an NER class depends on how many mentions of that class end up in the training split. In NER datasets, sentence counts and mention counts can look very different. Because of that, a split can satisfy a sentence-level constraint with zero error and still leave a class with too few training mentions for a model to learn it.

Specifically, DataSAIL's class-stratification constraint is:

Equation (1) ∑x∈Di∩Dcκ(x)≥(1−δ)⋅si⋅nc\sum_{x \in D_i \cap D^c} \kappa(x) \ge (1 - \delta) \cdot s_i \cdot n^c for every split ii and class cc

where:

  • DD is the set of data elements (sentences here), and DiD_i is the subset placed in split ii.
  • DcD^c is the subset of elements with at least one mention of class cc.
  • nc:=∑x∈Dcκ(x)n^c := \sum_{x \in D^c} \kappa(x), where κ(x)\kappa(x) is the weight of element xx (the number of samples in the cluster).
  • sis_i is the target size fraction for split ii, and δ\delta is the relative tolerance.

Notice that κ\kappa depends only on the element (κ:D→N≥1\kappa : D \to \mathbb{N}_{\ge 1}). It equals 11 for a single sentence and is >1> 1 only when xx is a cluster of sentences. The class cc only decides whether xx is in DcD^c. So one sentence gives the exact same weight κ(x)\kappa(x) to every class that appears in it: a sentence with 12 PER mentions and 1 ORG mention counts once toward PER and once toward ORG.

To balance mention counts, the weight has to depend on the (element, class) pair: κ(x,c)\kappa(x, c), the number of mentions of class cc in element xx. Putting these counts into the class matrix A∈N∣D∣×CA \in \mathbb{N}^{|D| \times C} with Ax,c:=κ(x,c)A_{x,c} := \kappa(x, c), our constraint is:

Equation (2) ∑x∈DiAx,c≥(1−δ)⋅si⋅nc\sum_{x \in D_i} A_{x,c} \ge (1 - \delta) \cdot s_i \cdot n^c, with nc:=∑x∈DAx,cn^c := \sum_{x \in D} A_{x,c} for every split ii and class cc

Here the sum runs over all x∈Dix \in D_i instead of Di∩DcD_i \cap D^c, because Ax,c=0A_{x,c} = 0 whenever xx has no mention of class cc.

In short, DataSAIL's constraint uses the 0/1 indicator [Ax,c>0][A_{x,c} > 0] weighted by κ(x)\kappa(x), while EMPIRE uses the full count matrix AA. Turning AA into 0/1 values throws away how many times each class appears in a sentence, and no choice of κ(x)\kappa(x) can bring those counts back because κ(x)\kappa(x) has no class index. The two constraints match only when every sentence has just one mention (when NER simplifies to sentence classification).

Here is a small 12-sentence example with two classes (PER, ORG) and an 8 / 4 train-test split (strain=2/3,stest=1/3s_{\text{train}} = 2/3, s_{\text{test}} = 1/3):

s1s_1s2s_2s3s_3s4s_4s5s_5s6s_6s7s_7s8s_8s9s_9s10s_{10}s11s_{11}s12s_{12}total
PER55544411111133
ORG11111155544433

Every sentence contains both classes, so nc=12n^c = 12 under Equation (1) and nc=33n^c = 33 under Equation (2). At δ=0\delta = 0, Equation (1) asks for ≥8\ge 8 train and ≥4\ge 4 test sentences per class, while Equation (2) asks for ≥22\ge 22 train and ≥11\ge 11 test mentions per class. Compare two splits:

  • Split A: train = {s1,…,s8}\{s_1, \dots, s_8\}
  • Split B: train = {s1,s2,s4,s5,s7,s8,s10,s11}\{s_1, s_2, s_4, s_5, s_7, s_8, s_{10}, s_{11}\}
Split ASplit B
Sentences (train / test)8 / 48 / 4
Equation (1) DataSAIL, PER (train / test)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)
Equation (1) DataSAIL, ORG (train / test)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)
PER mentions (train / test)29 / 422 / 11
ORG mentions (train / test)16 / 1722 / 11
Equation (2) EMPIRE, PER (train / test)29≥2229 \ge 22, 4≥114 \ge 11 fails (infeasible at δ=0\delta = 0)22≥2222 \ge 22, 11≥1111 \ge 11 (satisfied at δ=0\delta = 0)
Equation (2) EMPIRE, ORG (train / test)16≥2216 \ge 22 fails, 17≥1117 \ge 11 (infeasible at δ=0\delta = 0)22≥2222 \ge 22, 11≥1111 \ge 11 (satisfied at δ=0\delta = 0)
AuthorsOpenReview ↗

Response to reviewer TuZj (continued)

Both splits satisfy Equation (1) at δ=0\delta = 0 for both classes, so DataSAIL treats them as identical. Under Equation (2), Split A is infeasible: training gets only 16 of 33 ORG mentions (48% vs. the 67% target) and test gets only 4 of 33 PER mentions (12% vs. 33%). A model evaluated on Split A gets a low ORG score from weak training support, and its PER test score rests on just 4 mentions, unrelated to unseen test entities. That brings back the low-training-support problem from Section 5.4 even though Equation (1) reports zero error. Split B uses the same 8 / 4 sentence counts, is equally valid to DataSAIL, and satisfies Equation (2) at 22 / 11 mentions for both classes. EMPIRE rejects Split A and accepts Split B.

We also agree that MinCut's larger class drift does not by itself separate our objective from our class constraint. However, setting δ=1\delta = 1 turns off EMPIRE's class-balance constraint, giving a direct constraint-matched comparison with MinCut:

Entity disjointness (%) ↑Class drift (Hellinger) ↓Split sizes
Native84.460.0415exact
MinCut95.190.1403relaxed (AnatEM 0.50→0.510.50 \to 0.51)
EMPIRE (α=1\alpha = 1, δ=1\delta = 1)96.710.1407exact on 24 / 24
EMPIRE (α=1\alpha = 1, δ∗\delta^\ast)94.620.0179exact on 24 / 24

At δ=1\delta = 1, EMPIRE's class drift matches MinCut's (0.1407 vs. 0.1403, a difference of 0.0004), while EMPIRE reaches 1.52 points higher entity disjointness and keeps exact split sizes on all 24 datasets. Moving from δ=1→δ∗\delta = 1 \to \delta^\ast then shows the separate effect of the class constraint: drift drops from 0.1407 to 0.0179 (below Native's 0.0415) at a cost of 2.09 points of disjointness. We now present the comparison in these two steps in Section 5.1.

Action: We have replaced the inaccurate "one label per sample" claim in Sections 1–2 with the κ(x)\kappa(x) vs. Ax,cA_{x,c} comparison above (with the worked example in Appendix A.12), and added the matched δ=1\delta = 1 comparison with MinCut in Section 5.1.

Weakness-1(b) (Feasibility pre-check guarantee): We agree with the reviewer and have fixed Appendix A.2. Minimizing ϵ+∑tδt\epsilon + \sum_t \delta_t over a shared partition returns a feasible vector with minimum total slack, not separate per-class minimums (as the reviewer's A=(2,0,1,1)A = (2, 0, 1, 1), B=(2,1,1,0)B = (2, 1, 1, 0) example shows, no single split may hit every class's individual minimum at once). We have replaced the sentence claiming that each class's tolerance is driven to its own independent minimum.

The pre-check is still useful without that componentwise reading. Before the main solve, it checks whether a requested (ϵ,δ)(\epsilon, \delta) has any valid split for the dataset and KK, and returns a feasible tolerance vector when relaxation is needed. Because the main ILP runs for up to 7,200 s at α=1\alpha = 1 and 14,400 s at α<1\alpha < 1 (Appendix A.7), passing an infeasible target directly could spend the whole time budget and return nothing. It also shows when the requested 0.1 floor works as-is: 0.1 is returned unchanged for every class on 15 of 24 datasets at α=0\alpha = 0 and 14 of 24 at α=0.5\alpha = 0.5, and is relaxed only on the remaining datasets.

On ϵ\epsilon: we agree that putting ϵ\epsilon in the same sum trades split-size slack against class slack instead of strictly prioritizing exact split sizes, and we have stated this clearly in Appendix A.2. In practice, ϵ=0\epsilon = 0 was requested in every run and met on all 24 datasets at α=1\alpha = 1, while only 6 of 96 clustered cells in Table 2 differ at two decimals (all at α<1\alpha < 1, where whole clusters cannot be split apart). We have clarified in Appendix A.2 that this is an empirical result of our runs (Table 13), not a formal guarantee of the sum objective.

Action: We have (i) rewritten Appendix A.2 to state that the pre-check minimizes total slack ϵ+∑tδt\epsilon + \sum_t \delta_t and that its components are coupled instead of independent thresholds, and (ii) noted that δt∗\delta_t^\ast equals the 0.1 floor for every class on 15 / 24 (α=0\alpha = 0) and 14 / 24 (α=0.5\alpha = 0.5) datasets, leaving all reported configurations unchanged.

AuthorsOpenReview ↗

Response to reviewer TuZj (continued)

Weakness-1(c) (Clarifying enforced constraints):

Thank you for flagging these points. Here is what was run in each case and how we have updated the paper:

  • ϵ\epsilon: ϵ=0\epsilon = 0 was requested in every run. Whenever EMPIRE assigns individual sentences (α=1,K=0\alpha = 1, K = 0), the resulting ratios match Native on all 24 datasets. Under α<1\alpha < 1, clusters cannot be split apart, so split sizes change in whole-cluster steps and an exact match may not exist. Across Table 2, 6 of the 96 clustered cells differ at two decimals (all at α<1\alpha < 1 and δ∗\delta^\ast), compared with 0 of the 48 unclustered cells.

  • δ\delta and the pre-check: Our experiments use the pre-check output directly: δ∗\delta^\ast is enforced in the main solve, as stated in Section 4.3. The earlier note in Appendix A.2 saying the enforced values "are not necessarily equal to (ϵ∗,δ∗)(\epsilon^\ast, \delta^\ast)" and calling the pre-check "strictly as a diagnostic" described a more general workflow that we did not use, so we have removed it.

  • Table labels: Both labels referred to the exact same runs. Tables 3 and 9 called the constrained setting δ∗\delta^\ast (lb 0.1), while Tables 2, 5, 6 and 7 called it δ=0.1\delta = 0.1. All tables now use δ∗\delta^\ast (lb 0.1), and no run in the paper enforced a flat δ=0.1\delta = 0.1 across all classes. Table 8 having values up to 1.0 matches this: δt∗\delta_t^\ast is per class with a lower bound of 0.1 (equal to 0.1 for every class on 15 / 24 datasets at α=0\alpha = 0 and 14 / 24 at α=0.5\alpha = 0.5, and higher where 0.1 is infeasible).

  • Equation (4) vs. Figure 1: Equation (4), ∑lAtlclsXql≥(1−δt)PqNt\sum_l A_{tl}^{\text{cls}} X_{ql} \ge (1 - \delta_t) P_q N_t, sets a lower bound on each class's mention count in each split and does not force identical class ratios across splits. Figure 1's panel saying "Same class ratios in every split" overstated this. The constraint prevents splits from losing a class below that bound, and Table 5 reports the remaining Hellinger drift (which is above zero).

Action: We have (i) updated Section 4.3 to state that ϵ=0\epsilon = 0 is requested everywhere, met on all 24 datasets at α=1\alpha = 1, and reported realized ϵ∗\epsilon^\ast at α<1\alpha < 1, (ii) removed the "strictly diagnostic" remark in Appendix A.2 and stated that δ∗\delta^\ast is enforced, (iii) unified the δ∗\delta^\ast (lb 0.1) label across Tables 2–9, (iv) added Appendix A.14 (Table 13) with realized ϵ∗\epsilon^\ast and pre-check statuses alongside Table 8, and (v) updated Figure 1 and its caption so each class's mention share is bounded from below.

Weakness-1(d) (Optimization certificates and gaps): We agree with separating feasible solutions, proven optimal solutions, and structural impossibility, and we can break these out directly from our recorded solver logs.

The main ILP sets a relative MIP gap target of 0 for α=1\alpha = 1 and 5% for α<1\alpha < 1 (Appendix A.7). From Table 9, the final gaps are:

ConfigurationRequested toleranceMedian gapProven optimal (0.0%)Max gap
EMPIRE (α=1\alpha = 1, δ=1\delta = 1)00.0%13 / 24185.7% (FabNER)
EMPIRE (α=1\alpha = 1, δ∗\delta^\ast)09.9%6 / 24247.4% (FabNER)
EMPIRE (α=0\alpha = 0, δ=1\delta = 1)5%5.0%0 / 249.6%
EMPIRE (α=0\alpha = 0, δ∗\delta^\ast)5%4.9%2 / 2410.6%
EMPIRE (α=0.5\alpha = 0.5, δ=1\delta = 1)5%4.7%6 / 2477.5%
EMPIRE (α=0.5\alpha = 0.5, δ∗\delta^\ast)5%4.4%7 / 2463.4%

At α=1,δ=1\alpha = 1, \delta = 1 (our main entity-isolation setting), the median run is proven optimal (0.0% gap on 13 / 24 datasets). And for α<1\alpha < 1, the median gaps of 4.4–5.0% happen because the solver was configured to stop once it hit the 5% tolerance.

For structural limits, Table 10 checks the connected-component condition and shows that 100% disjointness at ϵ=0\epsilon = 0 is possible on 12 of the 24 datasets. On all 12, EMPIRE (α=1,δ=1\alpha = 1, \delta = 1) reaches 100% disjointness with a proven-optimal solution. On the other 12 datasets, 100% disjointness is impossible at ϵ=0\epsilon = 0 because of connected components in the data. We agree that this test only rules out 100% disjointness and does not bound how much partial overlap is unavoidable.

Reviewer_QoTiOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EMPIRE, which uses integer programming to create NER dataset splits with less entity overlap and context similarity while controlling split sizes and class balance. A parameter lets users trade off entity separation against context separation. Across 24 datasets, EMPIRE achieves entity separation comparable to MinCut with better class balance. Experiments with BiLSTM-CRF across all datasets and Llama-2-7B on five show that separating entities lowers F1, while reducing average context similarity has little effect. The authors also show that removing overlap without preserving class balance can leave some classes with very few training examples, causing additional F1 drops unrelated to recognizing unseen entities.

Critical strengths
  • The authors identify and address a confound in testing NER models on unseen entities. Preventing names from appearing in both training and test data can also leave some entity types poorly represented in training. A model may then fail on a new person’s name because it saw too few PERSON examples, rather than because it cannot recognize unfamiliar names. EMPIRE adds class-balance constraints to reduce this problem: the share of unseen test entities belonging to classes with fewer than ten training mentions falls from 5.8% to 0.1%. This makes the resulting evaluation easier to interpret by preserving training support for the entity types being tested.
  • The results show that better class balance is possible without sacrificing much entity separation. Across 24 datasets, the proposed balanced EMPIRE method improves entity disjointness over the original splits by 10.1 percentage points, compared with 10.7 for MinCut. Yet MinCut increases class-distribution drift, while EMPIRE reduces it. This gives the method a practical benefit: it creates challenging splits without introducing as much class imbalance, making the resulting scores easier to interpret.
Critical weaknesses
  • Novelty over existing methods is unclear: DataSAIL [1] already combines similarity-based splitting with class balancing. The paper instead compares against MinCut, a graph-partitioning heuristic for reducing entity overlap, which lacks the comparable class-balance constraints. This certainly shows that class balancing is useful but what is the contribution of EMPIRE beyond that?
  • How can authors conclude that context leakage has little effect when TopSim lowers F1 by 8.5 points while barely changing their metric? Could averaging similarity across all sentence pairs hide the near duplicates that actually help the model?

[1] Joeres, R., Blumenthal, D.B. & Kalinina, O.V. Data splitting to avoid information leakage with DataSAIL. Nat Commun 16, 3337 (2025).

Questions for authors
  • Section 4.3 says ε is fixed at zero, so split sizes should match the original dataset exactly. But Table 2 says ε can be increased when clustering makes this infeasible, and lists different ratios e.g. MIT movie’s validation share rises from 10% to 12%. Was ε fixed or adjusted for each dataset? Please report the values used, since the paper presents keeping split sizes unchanged as an advantage over MinCut.
5 comments
AuthorsOpenReview ↗

Response to Reviewer QoTi

We thank the reviewer for the helpful feedback, and especially for pointing out that our class-balance constraints make evaluation easier to interpret (cutting the share of unseen test entities in classes with fewer than 10 training mentions from 5.8% to 0.1%). We have uploaded a revised PDF with these updates in red.

Weakness-1: Thank you for raising this point. Below we explain how our class constraint differs from DataSAIL's and why NER requires that difference.

DataSAIL does allow multi-class labels and applies stratification to each class separately. The real difference is what the constraint counts. DataSAIL counts how many sentences contain a class, while learning an NER class depends on how many mentions of that class end up in the training split. In NER datasets, sentence counts and mention counts can look very different. Because of that, a split can satisfy a sentence-level constraint with zero error and still leave a class with too few training mentions for a model to learn it.

Specifically, DataSAIL's class-stratification constraint is:

Equation (1) ∑x∈Di∩Dcκ(x)≥(1−δ)⋅si⋅nc\sum_{x \in D_i \cap D^c} \kappa(x) \ge (1 - \delta) \cdot s_i \cdot n^c for every split ii and class cc

where:

  • DD is the set of data elements (sentences here), and DiD_i is the subset placed in split ii.
  • DcD^c is the subset of elements with at least one mention of class cc.
  • nc:=∑x∈Dcκ(x)n^c := \sum_{x \in D^c} \kappa(x), where κ(x)\kappa(x) is the weight of element xx (the number of samples in the cluster).
  • sis_i is the target size fraction for split ii, and δ\delta is the relative tolerance.

Notice that κ\kappa depends only on the element (κ:D→N≥1\kappa : D \to \mathbb{N}_{\ge 1}). It equals 11 for a single sentence and is >1> 1 only when xx is a cluster of sentences. The class cc only decides whether xx is in DcD^c. So one sentence gives the exact same weight κ(x)\kappa(x) to every class that appears in it: a sentence with 12 PER mentions and 1 ORG mention counts once toward PER and once toward ORG.

To balance mention counts, the weight has to depend on the (element, class) pair: κ(x,c)\kappa(x, c), the number of mentions of class cc in element xx. Putting these counts into the class matrix A∈N∣D∣×CA \in \mathbb{N}^{|D| \times C} with Ax,c:=κ(x,c)A_{x,c} := \kappa(x, c), our constraint is:

Equation (2) ∑x∈DiAx,c≥(1−δ)⋅si⋅nc\sum_{x \in D_i} A_{x,c} \ge (1 - \delta) \cdot s_i \cdot n^c, with nc:=∑x∈DAx,cn^c := \sum_{x \in D} A_{x,c} for every split ii and class cc

Here the sum runs over all x∈Dix \in D_i instead of Di∩DcD_i \cap D^c, because Ax,c=0A_{x,c} = 0 whenever xx has no mention of class cc.

In short, DataSAIL's constraint uses the 0/1 indicator [Ax,c>0][A_{x,c} > 0] weighted by κ(x)\kappa(x), while EMPIRE uses the full count matrix AA. Turning AA into 0/1 values throws away how many times each class appears in a sentence, and no choice of κ(x)\kappa(x) can bring those counts back because κ(x)\kappa(x) has no class index. The two constraints match only when every sentence has just one mention (when NER simplifies to sentence classification).

Here is a small 12-sentence example with two classes (PER, ORG) and an 8 / 4 train-test split (strain=2/3,stest=1/3s_{\text{train}} = 2/3, s_{\text{test}} = 1/3):

s1s_1s2s_2s3s_3s4s_4s5s_5s6s_6s7s_7s8s_8s9s_9s10s_{10}s11s_{11}s12s_{12}total
PER55544411111133
ORG11111155544433

Every sentence contains both classes, so nc=12n^c = 12 under Equation (1) and nc=33n^c = 33 under Equation (2). At δ=0\delta = 0, Equation (1) asks for ≥8\ge 8 train and ≥4\ge 4 test sentences per class, while Equation (2) asks for ≥22\ge 22 train and ≥11\ge 11 test mentions per class. Compare two splits:

  • Split A: train = {s1,…,s8}\{s_1, \dots, s_8\}
  • Split B: train = {s1,s2,s4,s5,s7,s8,s10,s11}\{s_1, s_2, s_4, s_5, s_7, s_8, s_{10}, s_{11}\}
Split ASplit B
Sentences (train / test)8 / 48 / 4
Equation (1) DataSAIL, PER (train / test)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)
Equation (1) DataSAIL, ORG (train / test)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)
PER mentions (train / test)29 / 422 / 11
ORG mentions (train / test)16 / 1722 / 11
Equation (2) EMPIRE, PER (train / test)29≥2229 \ge 22, 4≥114 \ge 11 fails (infeasible at δ=0\delta = 0)22≥2222 \ge 22, 11≥1111 \ge 11 (satisfied at δ=0\delta = 0)
Equation (2) EMPIRE, ORG (train / test)16≥2216 \ge 22 fails, 17≥1117 \ge 11 (infeasible at δ=0\delta = 0)22≥2222 \ge 22, 11≥1111 \ge 11 (satisfied at δ=0\delta = 0)
AuthorsOpenReview ↗

Response to reviewer QoTi (continued)

Both splits satisfy Equation (1) exactly at δ=0\delta = 0 for both classes, so DataSAIL cannot tell them apart. Under Equation (2), Split A fails: training gets only 16 of the 33 ORG mentions (48% vs. the 67% target), and test gets only 4 of the 33 PER mentions (12% vs. 33%). A model trained on Split A scores poorly on ORG simply because it saw too few ORG training examples, while its PER test score is computed on just 4 mentions, regardless of whether test entities were unseen. That brings back the exact low-training-support problem from Section 5.4 even though Equation (1) reports zero error. Split B uses the same 8 / 4 sentence counts, is equally valid to DataSAIL, and meets Equation (2) cleanly at 22 / 11 mentions for both classes. EMPIRE rejects Split A and accepts Split B.

More broadly, EMPIRE contributes two things beyond adding class balance to an existing splitter:

  1. A joint objective for both leakage types. DataSAIL reduces similarity-based leakage, MinCut reduces entity overlap, and TopSim reduces context similarity alone. EMPIRE optimizes entity isolation and context dissimilarity together in one integer program weighted by α\alpha. This lets us control the trade-off between both types of overlap under the same constraints, and lets us test which type inflates NER scores by sweeping α\alpha between 0 and 1.

  2. A mention-level class constraint. In NER, learning a class depends on how many mentions of it appear in training (Equation (2)), not how many sentences touch it. Section 5.4 shows that missing mention support drives much of the apparent F1 drop in leakage-reduced splits, and prior splitters (including DataSAIL) do not bound mention counts.

Action: We have (i) replaced the inaccurate "one label per sample" description of DataSAIL in Sections 1–2 with the κ(x)\kappa(x) vs. κ(x,c)\kappa(x, c) comparison above, and (ii) added this formal comparison and the 12-sentence worked example in Appendix A.12.

Weakness-2: Thank you for pointing this out. The difference comes from comparing runs with different constraints: TopSim uses no class-balance constraint, so the matching EMPIRE setting is α=0\alpha = 0 with δ=1\delta = 1, not α=0\alpha = 0 with δ∗\delta^\ast. Table 7 shows the mean paired BiLSTM-CRF test F1 change from Native across the 24 datasets:

ConfigurationClass-balance constraintΔF1\Delta\text{F1} vs. NativeLower on
TopSimnone−0.085-0.085 (p<0.001p < 0.001)21 / 24
EMPIRE (α=0,δ=1\alpha = 0, \delta = 1)none−0.128-0.128 (p<0.001p < 0.001)21 / 24
EMPIRE (α=0,δ∗\alpha = 0, \delta^\ast)enforced−0.009-0.009 (n.s.)12 / 24

When neither method enforces class balance, our context-only split lowers F1 by 0.128 compared with TopSim's 0.085 on the same 24 datasets and split sizes. The two EMPIRE rows differ because enforcing class balance (δ∗\delta^\ast) restricts which splits the solver can pick, which limits how much context similarity can be removed at α=0\alpha = 0 (Section 5.5). So the −0.009-0.009 number reflects the class-balanced setting, not context splitting in general.

We agree that our earlier wording in the abstract and Section 5.3 was too broad. Saying that removing context leakage alone does not lower F1 came only from EMPIRE (α=0,δ∗\alpha = 0, \delta^\ast). At δ=1\delta = 1, the same objective lowers F1 by 0.128 (p<0.001p < 0.001, lower on 21 of 24 datasets). We have narrowed this claim and reported both settings side by side.

Action: We have (i) updated the abstract, Section 5.3, and Section 6 to state that context-only splitting leaves F1 nearly unchanged under class-balance constraints (α=0,δ∗\alpha = 0, \delta^\ast: −0.009-0.009, n.s.) while lowering F1 without them (α=0,δ=1\alpha = 0, \delta = 1: −0.128-0.128, p<0.001p < 0.001), and (ii) added the matched-constraint comparison with TopSim (δ=1\delta = 1) in Section 5.3.

AuthorsOpenReview ↗

Response to reviewer QoTi (continued)

Question-1: ϵ\epsilon was fixed at 00 in every run in the paper. We never tuned or relaxed it per dataset. What changes across settings is whether ϵ=0\epsilon = 0 is possible, and that depends only on whether clustering is used:

  • α=1\alpha = 1 (K=0K = 0): EMPIRE assigns individual sentences. Because target split ratios come from the native split itself, an exact match always exists, and the feasibility pre-check confirms ϵ=0\epsilon = 0 on all 24 datasets before the main solve.
  • α<1\alpha < 1 (K=50K = 50): EMPIRE assigns whole clusters, and each cluster must go into a single split. Because clusters cannot be split apart, split sizes can only change by whole clusters, so matching the native ratio to two decimal places is sometimes impossible. When that happens, the pre-check returns the smallest feasible tolerance, which is at most the size of one cluster.

Counting cells in Table 2 whose actual split ratios differ from Native at two decimal places shows this clearly across all 24 datasets:

ConfigurationClusteringDatasets differing from Native
EMPIRE (α=1\alpha = 1, δ∗\delta^\ast)no (K=0K = 0)0 / 24
EMPIRE (α=1\alpha = 1, δ=1\delta = 1)no (K=0K = 0)0 / 24
EMPIRE (α=0\alpha = 0, δ=1\delta = 1)yes (K=50K = 50)0 / 24
EMPIRE (α=0.5\alpha = 0.5, δ=1\delta = 1)yes (K=50K = 50)0 / 24
EMPIRE (α=0\alpha = 0, δ∗\delta^\ast)yes (K=50K = 50)2 / 24
EMPIRE (α=0.5\alpha = 0.5, δ∗\delta^\ast)yes (K=50K = 50)4 / 24
MinCut-1 / 24
TopSim-0 / 24

All 6 differences appear in clustered (α<1\alpha < 1) columns when paired with the tight class tolerance δ∗\delta^\ast (6 of 96 clustered cells vs. 0 of 48 unclustered cells, and 0 clustered cells at δ=1\delta = 1). The mit-movie shift (0.70:0.10:0.20→0.69:0.12:0.200.70:0.10:0.20 \to 0.69:0.12:0.20) is in the clustered α=0.5,δ∗\alpha = 0.5, \delta^\ast cell, not the setting compared against MinCut. We have added Appendix A.14 (Table 13) listing the realized ϵ∗\epsilon^\ast and pre-check status across all runs (all 144 / 144 pre-checks closed to 0.0% MIP gap in at most 5.47 s, with ϵ∗=0\epsilon^\ast = 0 on 134 / 144 runs and 0<ϵ∗≤0.02020 < \epsilon^\ast \le 0.0202 on 10 clustered δ∗\delta^\ast runs).

For the comparison with MinCut: because MinCut targets entity overlap, we compare it against EMPIRE at α=1\alpha = 1 (K=0K = 0), which does not cluster and matches native split ratios on all 24 datasets. MinCut is the one whose split sizes move: on AnatEM it increases the training split from 0.50 to 0.51, which as Section 5 notes is how it reaches 100% disjointness there (while EMPIRE at α=1,ϵ=0\alpha = 1, \epsilon = 0 is not allowed to change split sizes).

Two places in our earlier draft caused this confusion, and we have fixed both:

  1. Section 4.3 said "We set ϵ=0\epsilon = 0 throughout, so every split matches the target ratio exactly." Setting ϵ=0\epsilon = 0 is true everywhere, but exact ratios are guaranteed only without clustering (α=1\alpha = 1).
  2. The Table 2 caption had "for α<0\alpha < 0, clustering with K=50K = 50 can make this infeasible", which was a typo for α<1\alpha < 1.

Action: We have (i) fixed the Table 2 caption typo from α<0\alpha < 0 to α<1\alpha < 1, (ii) updated Section 4.3 to state that ϵ=0\epsilon = 0 is requested in all runs and met on all 24 datasets at α=1\alpha = 1 (0<ϵ∗≤0.02020 < \epsilon^\ast \le 0.0202 on 10 / 96 clustered runs), (iii) reported realized ϵ∗\epsilon^\ast values and pre-check solve statuses per dataset in Appendix A.14 (Table 13), and (iv) stated clearly in Section 5.1 that the split-size comparison with MinCut is at α=1\alpha = 1, where ϵ∗=0\epsilon^\ast = 0 holds on all 24 datasets.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer Num2

We thank the reviewer for the helpful feedback, and especially for noting the value of our class-support analysis in showing why aggressive entity separation can unfairly lower NER scores. We have uploaded a revised PDF with all of these updates in red.

Weakness-1:

Thank you for raising this point. Below we explain how our class constraint differs from DataSAIL's and why NER needs that difference.

DataSAIL does allow multi-class labels and applies stratification to each class separately. The real difference is what the constraint counts. DataSAIL counts how many sentences contain a class, while learning an NER class depends on how many mentions of that class end up in the training split. In NER datasets, sentence counts and mention counts can look very different. Because of that, a split can satisfy a sentence-level constraint with zero error and still leave a class with too few training mentions for a model to learn it.

Specifically, DataSAIL's class-stratification constraint is:

Equation (1) ∑x∈Di∩Dcκ(x)≥(1−δ)⋅si⋅nc\sum_{x \in D_i \cap D^c} \kappa(x) \ge (1 - \delta) \cdot s_i \cdot n^c for every split ii and class cc

where:

  • DD is the set of data elements (sentences here), and DiD_i is the subset placed in split ii.
  • DcD^c is the subset of elements with at least one mention of class cc.
  • nc:=∑x∈Dcκ(x)n^c := \sum_{x \in D^c} \kappa(x), where κ(x)\kappa(x) is the weight of element xx (the number of samples in the cluster).
  • sis_i is the target size fraction for split ii, and δ\delta is the relative tolerance.

Notice that κ\kappa depends only on the element (κ:D→N≥1\kappa : D \to \mathbb{N}_{\ge 1}). It equals 11 for a single sentence and is >1> 1 only when xx is a cluster of sentences. The class cc only decides whether xx is in DcD^c. So one sentence gives the exact same weight κ(x)\kappa(x) to every class that appears in it: a sentence with 12 PER mentions and 1 ORG mention counts once toward PER and once toward ORG.

To balance mention counts, the weight has to depend on the (element, class) pair: κ(x,c)\kappa(x, c), the number of mentions of class cc in element xx. Putting these counts into the class matrix A∈N∣D∣×CA \in \mathbb{N}^{|D| \times C} with Ax,c:=κ(x,c)A_{x,c} := \kappa(x, c), our constraint is:

Equation (2) ∑x∈DiAx,c≥(1−δ)⋅si⋅nc\sum_{x \in D_i} A_{x,c} \ge (1 - \delta) \cdot s_i \cdot n^c, with nc:=∑x∈DAx,cn^c := \sum_{x \in D} A_{x,c} for every split ii and class cc

Here the sum runs over all x∈Dix \in D_i instead of Di∩DcD_i \cap D^c, because Ax,c=0A_{x,c} = 0 whenever xx has no mention of class cc.

In short, DataSAIL's constraint uses the 0/1 indicator [Ax,c>0][A_{x,c} > 0] weighted by κ(x)\kappa(x), while EMPIRE uses the full count matrix AA. Turning AA into 0/1 values throws away how many times each class appears in a sentence, and no choice of κ(x)\kappa(x) can bring those counts back because κ(x)\kappa(x) has no class index. The two constraints match only when every sentence has just one mention (when NER simplifies to sentence classification).

Here is a small 12-sentence example with two classes (PER, ORG) and an 8 / 4 train-test split (strain=2/3,stest=1/3s_{\text{train}} = 2/3, s_{\text{test}} = 1/3):

s1s_1s2s_2s3s_3s4s_4s5s_5s6s_6s7s_7s8s_8s9s_9s10s_{10}s11s_{11}s12s_{12}total
PER55544411111133
ORG11111155544433

Every sentence contains both classes, so nc=12n^c = 12 under Equation (1) and nc=33n^c = 33 under Equation (2). At δ=0\delta = 0, Equation (1) asks for ≥8\ge 8 train and ≥4\ge 4 test sentences per class, while Equation (2) asks for ≥22\ge 22 train and ≥11\ge 11 test mentions per class. Compare two splits:

  • Split A: train = {s1,…,s8}\{s_1, \dots, s_8\}
  • Split B: train = {s1,s2,s4,s5,s7,s8,s10,s11}\{s_1, s_2, s_4, s_5, s_7, s_8, s_{10}, s_{11}\}
Split ASplit B
Sentences (train / test)8 / 48 / 4
Equation (1) DataSAIL, PER (train / test)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)
Equation (1) DataSAIL, ORG (train / test)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)8≥88 \ge 8, 4≥44 \ge 4 (satisfied at δ=0\delta = 0)
PER mentions (train / test)29 / 422 / 11
ORG mentions (train / test)16 / 1722 / 11
Equation (2) EMPIRE, PER (train / test)29≥2229 \ge 22, 4≥114 \ge 11 fails (infeasible at δ=0\delta = 0)22≥2222 \ge 22, 11≥1111 \ge 11 (satisfied at δ=0\delta = 0)
Equation (2) EMPIRE, ORG (train / test)16≥2216 \ge 22 fails, 17≥1117 \ge 11 (infeasible at δ=0\delta = 0)22≥2222 \ge 22, 11≥1111 \ge 11 (satisfied at δ=0\delta = 0)
AuthorsOpenReview ↗

Response to Reviewer Num2 (continued)

Weakness-1 (continued): Both splits satisfy Equation (1) at δ=0\delta = 0 for both classes, so DataSAIL cannot tell them apart. Under Equation (2), Split A fails: training gets only 16 of the 33 ORG mentions (48% vs. the 67% target), and test gets only 4 of the 33 PER mentions (12% vs. 33%). A model evaluated on Split A gets a poor ORG score from weak training support, while its PER test score rests on just 4 mentions, unrelated to whether test entities are unseen. That brings back the low-training-support problem from Section 5.4 even though Equation (1) reports zero error. Split B uses the same 8 / 4 sentence counts, is equally valid to DataSAIL, and satisfies Equation (2) at 22 / 11 mentions for both classes. EMPIRE rejects Split A and accepts Split B.

Action: We have replaced the inaccurate "one label per sample" statement in Sections 1–2 with the κ(x)\kappa(x) vs. Ax,cA_{x,c} comparison above, and added the formal comparison and 12-sentence worked example in Appendix A.12.

Weakness-2: Thank you for raising this. The difference comes from comparing runs with different constraints: TopSim enforces no class-balance constraint, so its matching EMPIRE setting is α=0\alpha = 0 with δ=1\delta = 1, not α=0\alpha = 0 with δ∗\delta^\ast. Table 7 reports the mean paired BiLSTM-CRF test F1 change from Native across the 24 datasets:

ConfigurationClass-balance constraintΔF1\Delta\text{F1} vs. NativeLower on
TopSimnone−0.085-0.085 (p<0.001p < 0.001)21 / 24
EMPIRE (α=0\alpha = 0, δ=1\delta = 1)none−0.128-0.128 (p<0.001p < 0.001)21 / 24
EMPIRE (α=0\alpha = 0, δ∗\delta^\ast)enforced−0.009-0.009 (n.s.)12 / 24

Without class-balance constraints, our context-only split lowers F1 by 0.128 compared with TopSim's 0.085 on the same datasets and split sizes. Meanwhile, the −0.009-0.009 number happens under the δ∗\delta^\ast class constraint, which restricts which splits the solver can pick and limits how much context similarity can be removed (Section 5.5).

As for why our similarity metric barely changes under TopSim: EMPIRE minimizes IL=∑l1<l2Wl1,l2⋅vl1⋅vl2⋅Zl1,l2I_L = \sum_{l_1 < l_2} W_{l_1,l_2} \cdot v_{l_1} \cdot v_{l_2} \cdot Z_{l_1,l_2} over cluster centroids and assigns whole clusters together. Near-duplicate sentences inside one cluster cannot be separated, and near-duplicates across two clusters only affect their cluster averages. Our reported evaluation metric (the average cosine similarity over all cross-split sentence pairs) also averages over all pairs, so neither quantity reacts strongly to a small number of near-duplicate pairs. That matches what Section 5.3 and Appendix A.6 note for TopSim: it removes near-duplicate contexts, lowers F1 by 0.085, and changes mean information leakage by only 0.013.

In the revised paper, we state that reducing mean cross-split centroid similarity under fixed split sizes and δ∗\delta^\ast class constraints leaves F1 statistically unchanged, whereas removing near-duplicates (TopSim) or removing class constraints (δ=1\delta = 1) lowers F1, and note that our metric does not capture rare near-duplicate overlap.

Action: We have (i) replaced the broad claim in the abstract, Section 5.3, and Section 6 with this narrower statement, (ii) reported the δ=1\delta = 1 context result next to the δ∗\delta^\ast one wherever that claim is made, and (iii) noted in Sections 3.3, 4.5, and Appendix A.11 that the cluster-level objective has no term sensitive to individual near-duplicate pairs.

AuthorsOpenReview ↗

Response to Reviewer Num2 (continued)

Weakness-3: Thank you for flagging both points. We clarify each below.

First, on ϵ\epsilon and split sizes: ϵ=0\epsilon = 0 was requested in every run and never tuned per dataset. Whenever EMPIRE assigns individual sentences (α=1,K=0\alpha = 1, K = 0), the resulting split ratios match Native on all 24 datasets. Under α<1\alpha < 1, whole clusters must go to the same split, so split sizes can only move in whole-cluster steps and an exact two-decimal match may not exist. Across Table 2, 6 of the 96 clustered cells differ from Native at two decimals (all at α<1\alpha < 1 and δ∗\delta^\ast, vs. 0 of 48 unclustered cells): CrossNER_science and MAPA_fine at α=0\alpha = 0, and CrossNER_music, CrossNER_science, mit-movie, and StackOverflow-NER at α=0.5\alpha = 0.5. We have updated Section 4.3 ("so every split matches the target ratio exactly") so it distinguishes the requested ϵ=0\epsilon = 0 from realized ϵ∗\epsilon^\ast in clustered runs (reported per dataset in Appendix A.14, Table 13). Also, our split-size comparison with MinCut is at α=1\alpha = 1 (K=0K = 0), which matches Native on all 24 datasets, whereas MinCut shifts AnatEM's training split from 0.50 to 0.51 to reach 100% disjointness (Section 5).

Second, on Appendix A.2: we agree with the reviewer and have updated Appendix A.2. Minimizing ϵ+∑tδt\epsilon + \sum_t \delta_t over a shared partition returns a feasible tolerance vector with minimum total slack, not separate per-class minimums, since all terms share the same partition variables. We have updated Appendix A.2's statement on infeasibility below (ϵ∗,δ∗)(\epsilon^\ast, \delta^\ast) so that it refers to total slack instead of separate coordinates. None of our reported results depend on this wording: δ∗\delta^\ast is simply passed as the enforced tolerance to the main solve, Table 5 measures actual Hellinger class drift on the final splits, and δt∗\delta_t^\ast equals the requested 0.1 floor for every class on 15 of 24 datasets at α=0\alpha = 0 and 14 of 24 at α=0.5\alpha = 0.5 (Table 8).

Action: We have (i) rewritten Section 4.3 to state that ϵ=0\epsilon = 0 is requested throughout and met on all 24 datasets at α=1\alpha = 1 (0<ϵ∗≤0.02020 < \epsilon^\ast \le 0.0202 on 10 / 96 clustered runs), (ii) added Appendix A.14 (Table 13) reporting realized ϵ∗\epsilon^\ast and pre-check optimality across all runs, (iii) rewritten Appendix A.2 so the pre-check is described as minimizing total slack with coupled components, and (iv) stated clearly in Section 5.1 that the split-size comparison with MinCut is made at α=1\alpha = 1, where ϵ∗=0\epsilon^\ast = 0 holds on all 24 datasets.

Question-1: What prevents a fair head-to-head run with DataSAIL is its class-balance constraint. As shown in Weakness-1, DataSAIL's weight κ(x)\kappa(x) has no class index, so it can only bound how many sentences touch each class, not how many mentions each class gets. Running EMPIRE (mention-balanced) against DataSAIL (sentence-balanced) would measure the gap between those two constraints instead of comparing the solvers under equal class support.

Your summary of the methodological difference is spot on: EMPIRE combines mention-count class bounds with an entity-isolation objective, and NER needs both together. DataSAIL only minimizes cross-split similarity and has nothing forcing an entity into a single split. At the same time, isolating entities tends to pack mentions of a class together, so mention-level bounds are needed to keep splits from losing classes, which Section 5.4 shows would otherwise cause a misleading F1 drop.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.