acceptodds
Under review as a conference paper at ICLR 2027

ProFiNER: Progressive Entity Calibration and Conflict-Aware Inference for Zero-Shot Named Entity Recognition

Abstract

Zero-shot named entity recognition without task-specific annotations provides a flexible solution for adapting entity extraction to new domains and label spaces. However, direct prompting with large language models often suffers from incomplete entity discovery, inaccurate boundaries, type confusion, and inconsistent predictions among competing candidates. To address these distinct failure modes systematically, we propose ProFiNER, a progressive inference framework for robust zero-shot NER. Specifically, ProFiNER first integrates type-agnostic mention mining with decomposed type-wise question answering and iterative sampling to build a high-recall candidate entity pool. It then selectively refines uncertain entity boundaries and calibrates candidate types through label-wise score normalization and pairwise comparison in an extended label space containing an explicit non-entity category. Finally, structurally incompatible candidates are represented by a conflict graph and jointly resolved through evidence-based adjudication and global selection. Experiments on multiple NER benchmarks with different language model backbones demonstrate that ProFiNER consistently improves entity recognition over direct prompting. Further ablation analyses demonstrate that our type calibration mechanism effectively alleviates semantic confusion, while conflict-aware global inference significantly enhances the structural consistency of the final predictions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Zero-shot Named Entity Recognition, Large Language Models, Progressive Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZHvnOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProFiNER, a progressive framework for zero-shot NER that combines candidate mining, boundary calibration, type calibration, and conflict-aware selection. Experiments across multiple NER benchmarks and LLM backbones demonstrate consistent improvements over direct prompting, supported by component ablation studies.

Critical strengths

The method achieves consistent improvements across multiple benchmarks and LLM backbones. The ablation studies further demonstrate the effectiveness of its major components.

Critical weaknesses
  1. The proposed framework introduces substantial inference overhead, requiring 5.58–13.82 model requests per sentence, as reported in Table 6. However, the paper lacks comparisons with computationally matched baselines and does not report inference latency or token costs. This makes it difficult to assess whether the performance gains justify the additional computational expense.

  2. The type calibration mechanism relies on statistics computed over the entire candidate pool (Equations 6–7). This introduces dependencies across evaluation samples, potentially limiting applicability to independent or streaming inference.

Questions for authors
  1. Can the authors provide comparisons with runtime cost?

  2. How does the type calibration mechanism perform when statistics are computed from smaller batches or an independent unlabeled calibration set?

  3. How are the hyperparameters in Appendix A selected?

5 comments

Response to W1 and Q1: Runtime Cost Comparison and Stage-wise Computational Cost Analysis

Thank you for this valuable comment. We agree that runtime cost is important for evaluating the practical applicability of ProFiNER. We therefore provide same-backbone comparisons with existing tuning-free NER methods and a stage-wise computational cost analysis.

(1) Runtime Comparison with Existing Tuning-free NER Methods

We evaluate all methods using the same Qwen2.5-32B backbone, including Direct Prompting, Self-Consistency (K=5 and K=10), Self-Improving, and EvoPrompt.

The results are summarized below.

MethodAvg. F1 ↑Input tokens/sentence ↓Output tokens/sentence ↓Latency/sentence ↓GPU·h ↓
Direct Prompting72.281313.1143.860.1301 s0.5628
Self-Consistency (K=5)72.921313.11225.480.3573 s1.5457
Self-Consistency (K=10)72.141313.11451.160.7067 s3.0569
Self-Improving70.61956.5255.151.1020 s4.7668
EvoPrompt74.574482.0577.131.3683 s5.9184
ProFiNER77.542739.61481.061.0804 s4.6732

ProFiNER achieves the highest average F1 (77.54%). Compared with Self-Improving, it improves F1 by 6.93 percentage points (77.54% vs. 70.61%), while requiring slightly less inference time (1.0804 vs. 1.1020 s/sentence) and GPU consumption (4.6732 vs. 4.7668 GPU-hours), despite higher token consumption.

Compared with EvoPrompt, ProFiNER improves F1 by 2.97 percentage points and reduces input tokens, inference latency, and GPU consumption by approximately 38.9%, 21.0%, and 21.0%, respectively. Although its output-token consumption is higher (481.06 vs. 77.13), it achieves better extraction performance with lower overall measured computational costs.

Compared with Self-Consistency (K=10), ProFiNER incurs higher computational costs (1.0804 vs. 0.7067 s/sentence; 4.6732 vs. 3.0569 GPU-hours), but improves F1 by 5.40 percentage points. This illustrates the trade-off between computational overhead and extraction performance.

(2) Stage-wise Computational Cost Analysis

We further analyze the computational costs of ProFiNER's four stages. The results are presented below.

StageInput tokens/sentenceOutput tokens/sentenceLatency/sentenceGPU·hShare of total time
Dual Candidate Mining2100.00358.200.6983 s3.020664.64%
Boundary-aware Calibration43.966.070.0883 s0.38198.17%
Type-aware Calibration430.97110.430.2568 s1.110723.77%
Conflict-aware Selection164.686.360.0370 s0.16003.42%
Four-stage total2739.61481.061.0804 s4.6732100%

Dual Candidate Mining contributes the largest computational cost (64.64%, 3.0206 GPU-hours), followed by Type-aware Calibration (23.77%, 1.1107 GPU-hours). Boundary-aware Calibration and Conflict-aware Selection account for 8.17% and 3.42%, respectively. Together, the first two stages account for 88.41% of the total computational cost, identifying the primary optimization targets.

The complete framework consumes 2,739.61 input tokens and 481.06 output tokens per sentence, with an average latency of 1.0804 seconds and a total cost of 4.6732 GPU-hours.

Overall, same-backbone comparisons demonstrate that ProFiNER achieves the highest extraction performance among the evaluated methods while maintaining competitive efficiency relative to computationally intensive baselines. The stage-wise analysis further identifies the main computational bottlenecks and directions for future optimization

Response to W2 and Q2: Robustness of Type-aware Calibration under Different Statistical Estimation Scopes

Thank you for raising this important concern. We agree that investigating the dependence of Type-aware Calibration on the entire test-set candidate pool is important for assessing its applicability to independent and streaming inference. To address this concern, we conduct additional experiments using different statistical estimation scopes, including the full test-set candidate pool, mini-batches of different sizes (B=8, 32, and 128), and per-sentence estimation.

(1) Mini-batch Statistical Estimation.

In ProFiNER, Type-aware Calibration employs Per-Type Score Standardization to account for differences in score distributions across entity types, facilitating subsequent pairwise adjudication. To investigate whether this process depends on global statistics, we vary the scope of candidate scores used to calculate per-type means and standard deviations.

The experimental results are summarized below.

Calibration SettingCoNLL03ΔMovieΔRestaurantΔWikiGoldΔ
Full test set82.53–75.45–72.24–79.00–
Mini-batch (B=8)81.99−0.5475.15−0.3072.26+0.0278.99−0.01
Mini-batch (B=32)82.04−0.4975.19−0.2672.20−0.0478.99−0.01
Mini-batch (B=128)82.39−0.1475.29−0.1672.240.0078.99−0.01
Per-sentence82.43−0.1075.71+0.2672.28+0.0479.11+0.11

The results indicate that Type-aware Calibration maintains stable performance under different statistical estimation scopes. With B=8, F1 decreases by only 0.54, 0.30, and 0.01 percentage points on CoNLL03, Movie, and WikiGold, respectively, while Restaurant shows a slight improvement of 0.02 percentage points. As the batch size increases to 128, performance becomes even closer to full-test-set estimation, with changes of −0.14, −0.16, 0.00, and −0.01 percentage points across the four datasets. These results suggest that estimating per-type statistics from smaller candidate pools does not substantially affect extraction performance.

(2) Per-sentence Statistical Estimation.

We further investigate a per-sentence setting to determine whether Type-aware Calibration can operate independently without relying on statistics from other evaluation samples. For each sentence, we independently calculate the mean and standard deviation of each entity type using only the raw candidate scores within that sentence. The resulting scores are then standardized and passed to the same pairwise type adjudication and conflict-aware selection procedures. If an entity type has only one candidate within a sentence, its standardized score is set to zero.

For a controlled comparison on CoNLL03, we fix the 7,093 second-stage candidates generated in the original experiment and reuse their raw type scores, changing only the scope of statistical estimation. The F1 score decreases marginally from 82.5346% under full-test-set estimation to 82.4276% under per-sentence estimation, corresponding to a difference of only 0.1070 percentage points. This demonstrates that local statistical estimation can achieve performance highly comparable to global estimation.

Overall, these findings demonstrate that Per-Type Score Standardization serves as a supporting mechanism for Type-aware Calibration, while global statistics are not essential to maintaining its extraction performance. The comparable results under mini-batch and per-sentence settings indicate that ProFiNER can accommodate different statistical estimation scopes without requiring access to the entire test-set candidate pool. This provides empirical evidence of its applicability to batch-based and sentence-level inference scenarios.

Response to Q3: Hyperparameter Selection and Sensitivity Analysis

Thank you for this constructive suggestion. We clarify our hyperparameter selection strategy and provide additional sensitivity analyses of the scoring weights and sampling parameters to evaluate the robustness of ProFiNER.

1. Hyperparameter Selection Strategy.

The hyperparameters are selected based on preliminary experimental observations and the functional roles of individual components, aiming to balance candidate coverage, calibration reliability, and computational efficiency.

The original manuscript uniformly set K₁=K₂=10 across all datasets. As this conservative configuration may introduce unnecessary inference overhead, we revised the sampling parameters based on dataset-specific sensitivity results.

For the final scoring function, we retain consistent weighting coefficients across all four datasets for five evidence sources: cluster support (λCλ_C), boundary evidence (λBλ_B), initial type confidence (λTλ_T), calibrated type evidence (λϕ\lambda_\phi), and adjudication margin (λmλ_m).

WeightValue
λCλ_C0.175
λBλ_B0.500
λTλ_T0.175
λϕ\lambda_\phi0.105
λmλ_m0.505

The revised dataset-specific sampling parameters and calibration thresholds are:

DatasetK₁K₂θm\theta_mθt\theta_tθb\theta_b
CoNLL03310.300.400.00
Movie350.000.500.00
WikiGold530.570.500.50
Restaurant320.400.350.40

Scoring weights are shared across datasets, whereas sampling parameters and calibration thresholds are adapted to dataset characteristics.

2. Sensitivity Analysis.

The corresponding sensitivity-analysis figures are provided in Appendix F of the revised manuscript.

(a) Sensitivity Analysis of Scoring Weights.

We conduct one-at-a-time sensitivity analyses by varying each of the five weighting coefficients while keeping all other coefficients and evidence inputs fixed. Across the evaluated parameter ranges, final entity-level F1 varies by no more than approximately 0.5 percentage points on all four datasets. This indicates that ProFiNER is relatively insensitive to moderate changes in scoring weights, supporting the robustness of its evidence integration mechanism.

(b) Sensitivity Analysis of Sampling Hyperparameters.

We vary K₁ and K₂ from 1 to 10 and evaluate their effects on initial candidate boundary recall across the four datasets. To isolate the effects of sampling configurations, all experiments use a uniform boundary threshold of θbθ_b=0.

As shown in Appendix F, CoNLL03 and WikiGold exhibit relatively stable recall across different sampling configurations. Movie benefits more noticeably from increasing K₂, while Restaurant shows a more moderate improvement. These results indicate that the optimal sampling budget varies across datasets.

Based on these observations, we replace the original K₁=K₂=10 configuration with the dataset-specific settings presented above. Under the same θ_b=0 evaluation, candidate boundary recall decreases by only 0.5, 0.9, 0.4, and 0.8 percentage points on CoNLL03, Movie, WikiGold, and Restaurant, respectively.

Meanwhile, K₁+K₂ decreases from 20 to 4, 8, 8, and 5, corresponding to reductions of 80%, 60%, 60%, and 75%.

Overall, final entity-level F1 remains stable under moderate variations in scoring weights, while the revised dataset-specific sampling configurations maintain candidate boundary recall close to that of the original setting. These findings support our hyperparameter selection strategy and demonstrate the robustness of ProFiNER.

Reviewer_KrYUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ProFiNER, a progressive inference framework for zero-shot NER. It addresses four interrelated major challenges in zero-shot NER: incomplete mention detection, inaccurate boundaries, entity type confusion, and structural inconsistency. The framework integrates dual candidate mining, boundary-aware calibration, type-aware calibration, and conflict-aware selection into a four-stage pipeline. Experiments across multiple NER benchmarks and three LLM backbones show consistent improvements over same-backbone direct prompting.

Critical strengths

ProFiNER consistently outperforms direct prompting across multiple NER benchmarks and LLM backbones. Furthermore, it achieves superior performance compared to existing NER baselines on most evaluated benchmarks.

Critical weaknesses

The empirical evaluation does not sufficiently establish the benefit of the proposed pipeline relative to its additional complexity and computational cost. This concern is supported by three limitations:

  1. Baseline comparisons. Same-backbone comparisons with direct prompting and component ablations demonstrate improvements, but comparisons with stronger methods such as EvoPrompt, DiZiNER, and JPT are not conducted under a shared-backbone setting. A cost-matched repeated-sampling baseline with span-level voting is also missing, leaving unclear whether the proposed pipeline is preferable to simpler uses of additional inference computation.

  2. Cost analysis. Appendix D reports logical request counts, which do not capture the full generation workload: Stage 1 alone generates 10(1+∣L∣)10(1+|L|) samples per sentence. Without token counts and runtime comparisons against direct prompting and representative stronger baselines such as DiZiNER or JPT, the accuracy–cost trade-off remains unclear.

  3. Label-space scalability. The evaluated benchmarks have relatively small label inventories, such as CoNLL-2003 (4 types) and MIT-Restaurant (8 types). Although CrossNER adds domain-specific fine-grained evaluation, scalability to larger inventories remains unverified. Given the type-wise sampling and per-label calibration, evaluation on a benchmark such as Few-NERD (66 fine-grained types) would be particularly informative.

Together, these limitations make it difficult to determine whether the framework offers sufficient benefits over strong alternatives to justify its additional inference complexity, particularly as the label space grows.

Questions for authors
  1. Controlled Comparisons: Could the authors provide comparisons with existing tuning-free NER methods using the same LLM backbones? Additionally, have cost-matched baselines, such as repeated sampling with majority voting under comparable inference budgets, been evaluated? These comparisons would help clarify whether the reported improvements are attributable to the proposed framework rather than backbone differences or additional test-time computation.
  2. Computational Cost: Could the authors provide a more detailed inference costs, including input/output token counts and GPU runtime for each stage? In particular, how do these costs compare with those of existing NER baselines? This information is important for assessing whether the performance gains justify the additional computational overhead.
  3. Scalability to Larger Entity Schemas: How do the computational cost and extraction performance of ProFiNER change as the number of entity types increases? Could the authors provide evidence of scalability on datasets with substantially larger label spaces, such as Few-NERD, or discuss any observed limitations of the type-wise candidate mining and label-wise calibration stages?
  4. Hyperparameter Search and Sensitivity: Could the authors clarify how the hyperparameters were selected, particularly the final scoring weights? Additionally, a sensitivity analysis of key hyperparameters would help assess the robustness of the proposed framework.
  5. Minor Clarifications and Additional Analysis:
    • Cluster Score for Corrected Spans: How is the cluster-level score sclusters_{\text{cluster}} assigned to newly introduced spans during Boundary-aware Calibration? Do corrected spans inherit the original cluster score, or is it recomputed?
    • Boundary-Aware Calibration Analysis: Could the authors provide a more detailed analysis of how often Boundary-Aware Calibration successfully corrects incorrect boundaries versus incorrectly modifies originally correct ones? It would also be useful to examine how these corrections affect entity type assignments and final entity-level predictions.
8 comments

Response to W1 and Q1: Controlled Comparisons with Existing Methods and Computational Cost Analysis

Thank you for the constructive and insightful comments. We agree that a comprehensive evaluation should establish whether the performance improvements of ProFiNER justify its additional computational complexity. To address this concern, we have supplemented controlled comparisons using the same LLM backbone and conducted a more detailed analysis of inference costs, including token consumption, inference latency, and GPU-hours.

1. Controlled Comparisons Using the Same LLM Backbone.

To eliminate the influence of backbone differences, we conduct comparisons with several representative tuning-free NER methods using the same Qwen2.5-32B backbone. These methods include Direct Prompting, Self-Consistency (K=5 and K=10), Self-Improving, and EvoPrompt.

The experimental results, including average F1 scores and computational costs, are summarized in the following table.

MethodAvg. F1 ↑Input tokens/sentence ↓Output tokens/sentence ↓Latency/sentence ↓GPU-hours ↓
Direct Prompting72.281313.1143.860.1301 s0.563
Self-Consistency (K=5)72.921313.11225.480.3573 s1.546
Self-Consistency (K=10)72.141313.11451.160.7067 s3.057
Self-Improving70.61956.5255.151.1020 s4.767
EvoPrompt74.574482.0577.131.3683 s5.918
ProFiNER77.542739.61481.071.0804 s4.677

As shown in the following table, ProFiNER achieves the highest average F1 score of 77.54%, outperforming Direct Prompting, Self-Consistency (K=5 and K=10), Self-Improving, and EvoPrompt by 5.26, 4.62, 5.40, 6.93, and 2.97 percentage points, respectively. These results demonstrate that ProFiNER achieves superior extraction performance under the same backbone, rather than benefiting from a stronger underlying LLM.

2. Comparison of Inference Costs and Performance–Efficiency Trade-offs.

We further analyze the computational costs to investigate whether ProFiNER's performance improvements justify its additional inference overhead. In particular, we compare its efficiency with Self-Improving and EvoPrompt.

Compared with Self-Improving, ProFiNER achieves a substantially higher average F1 score (77.54% vs. 70.61%), representing a 6.93-percentage-point improvement. Meanwhile, their inference latencies (1.0804 s vs. 1.1020 s per sentence) and GPU resource consumption (4.677 vs. 4.767 GPU-hours) are comparable. Although ProFiNER consumes more input and output tokens, it achieves considerably better extraction performance with slightly lower measured latency and GPU-hour consumption.

Compared with EvoPrompt, ProFiNER improves average F1 by 2.97 percentage points (77.54% vs. 74.57%). More importantly, it reduces input-token consumption by approximately 38.9%, inference latency by 21.0%, and GPU resource consumption by 21.0%. Although ProFiNER generates more output tokens (481.07 vs. 77.13), it achieves substantially better extraction performance while requiring fewer input tokens and less overall measured inference time and GPU resources.

Furthermore, the Self-Consistency experiments show that increasing the sampling number from K=5 to K=10 does not necessarily improve extraction performance (72.92% vs. 72.14%), despite increasing output-token consumption and computational costs. This suggests that simply allocating more inference computation to repeated sampling does not guarantee better NER performance.

Overall, these results demonstrate that ProFiNER achieves the highest average F1 among the evaluated methods while maintaining competitive computational costs. In particular, it provides substantially better extraction performance than Self-Improving under comparable inference costs and outperforms EvoPrompt in both extraction performance and measured inference efficiency. These findings highlight the effectiveness of ProFiNER's framework design and demonstrate its favorable performance–efficiency trade-off.

Response to W2 and Q2: Stage-wise Computational Cost Analysis and Comparison with Existing NER Baselines

Thank you for this valuable comment. We agree that a detailed analysis of computational costs is essential for understanding the practical applicability of ProFiNER. To address this concern, we provide a comprehensive breakdown of input/output token consumption, inference latency, and GPU-hours for each stage. We also compare the computational costs of ProFiNER with existing NER baselines to evaluate whether its performance improvements justify the additional computational overhead.

1. Stage-wise Inference Cost Analysis.

We first analyze the computational costs of the four stages of ProFiNER, including Dual Candidate Mining, Boundary-aware Calibration, Type-aware Calibration, and Conflict-aware Selection. Specifically, we report the average input/output tokens and inference latency per sentence, together with the corresponding GPU-hours. The results are summarized in the following table.

StageInput tokens/sentenceOutput tokens/sentenceLatency/sentenceGPU-hoursShare of total cost
Dual Candidate Mining2100.00358.200.6983 s3.020664.64%
Boundary-aware Calibration43.966.070.0883 s0.38198.17%
Type-aware Calibration430.97110.430.2568 s1.110723.77%
Conflict-aware Selection164.686.360.0370 s0.16003.42%
Four-stage total2739.61481.061.0804 s4.6732100%

As shown in the table, the complete ProFiNER framework consumes an average of 2,739.61 input tokens and 481.06 output tokens per sentence, with an average inference latency of 1.0804 seconds and a total computational cost of 4.6732 GPU-hours.

The stage-wise cost analysis shows that Dual Candidate Mining accounts for the largest proportion of the overall computational cost (64.64%), followed by Type-aware Calibration (23.77%). Boundary-aware Calibration and Conflict-aware Selection account for 8.17% and 3.42%, respectively.

2. Comparison of Computational Costs with Existing NER Baselines.

To further assess the computational efficiency of ProFiNER, we compare its inference costs with those of existing tuning-free NER baselines using the same Qwen2.5-32B backbone. The compared methods include Direct Prompting, Self-Consistency (K=5 and K=10), Self-Improving, and EvoPrompt. The detailed comparisons of average F1, input/output token consumption, inference latency, and GPU-hours are provided in our response to W1 and Q1.

Compared with Self-Improving, ProFiNER achieves a 6.93-percentage-point improvement in average F1 (77.54% vs. 70.61%), while exhibiting comparable inference latency (1.0804 s vs. 1.1020 s per sentence) and GPU consumption (4.6732 vs. 4.7668 GPU-hours). Although ProFiNER consumes more input and output tokens, it achieves substantially better extraction performance with slightly lower inference latency and GPU consumption.

Compared with EvoPrompt, ProFiNER improves average F1 by 2.97 percentage points (77.54% vs. 74.57%). More importantly, it reduces input-token consumption by approximately 38.9%, inference latency by approximately 21.0%, and GPU consumption by approximately 21.0%. Although ProFiNER generates more output tokens (481.06 vs. 77.13 per sentence), it achieves better extraction performance with fewer input tokens and lower overall measured inference costs.

Compared with relatively lightweight approaches such as Self-Consistency (K=10), ProFiNER incurs higher computational costs, requiring 4.6732 GPU-hours compared with 3.0569 GPU-hours and an inference latency of 1.0804 seconds compared with 0.7067 seconds per sentence. However, this additional computational overhead is accompanied by a substantial improvement in extraction performance, with average F1 increasing by 5.40 percentage points (77.54% vs. 72.14%).

Overall, these comparisons demonstrate that ProFiNER achieves a favorable performance–efficiency trade-off among the evaluated methods.

Response to W3 and Q3: Scalability Evaluation on Larger Entity Label Spaces

Thank you for raising this important question regarding the scalability of ProFiNER to larger entity schemas. To investigate this issue, we conducted additional experiments on Few-NERD, which contains 66 fine-grained entity types. Due to the substantial inference cost, we randomly sampled 2,000 sentences from the Few-NERD test set, containing 5,150 gold entities, for evaluation. We used Qwen2.5-32B as the backbone and fixed the sampling hyperparameters (K₁=K₂=4) across both the eight-type coarse-grained and 66-type fine-grained settings to ensure a controlled comparison. The average inference time per sentence was calculated by dividing the total elapsed time by the 2,000 evaluated sentences.

The experimental results are summarized in the following table.

Number of entity typesExact entity F1 (%)F1 after mapping to 8 coarse types (%)Boundary F1 (%)Input tokens/sentenceOutput tokens/sentenceTime/sentence
867.1867.1876.662,5189712.618 s
6654.1568.8676.4916,0912,5648.119 s

1. Scalability of Extraction Performance.

As the number of entity types increases from 8 to 66, the exact entity F1 decreases from 67.18% to 54.15%, reflecting the increased difficulty of distinguishing among a larger number of fine-grained entity types. To further investigate the source of this performance degradation, we map the 66 fine-grained entity types to their corresponding categories in the eight-type coarse-grained schema and evaluate the resulting predictions under the unified label space.

Interestingly, the mapped F1 reaches 68.86%, slightly higher than the 67.18% obtained directly under the eight-type setting. Meanwhile, the boundary F1 remains nearly unchanged (76.49% vs. 76.66%), with a difference of only 0.17 percentage points. These results suggest that the performance degradation primarily arises from fine-grained entity type disambiguation rather than entity boundary detection. In other words, ProFiNER maintains stable boundary extraction and coarse-grained recognition even when operating over a substantially larger label space.

2. Scalability of Computational Costs.

We further investigate how the increased number of entity types affects computational costs. As the label space expands from 8 to 66 types (an 8.25-fold increase), input-token consumption increases by approximately 6.39 times (2,518 to 16,091 tokens per sentence), output-token consumption increases by approximately 2.64 times (971 to 2,564 tokens), and average inference time increases by approximately 3.10 times (2.618 to 8.119 seconds per sentence).

To further identify the sources of this overhead, we examine the computational costs of two key stages.

Type-wise Candidate Mining is the primary contributor to the additional computational overhead. The total number of requests increases from 18,000 in the eight-type setting to 134,000 in the 66-type setting, representing an approximately 7.44-fold increase. This stage accounts for 77.2% of the total elapsed time in the 66-type setting. The increased workload results from the additional candidate-generation and evaluation operations required to explore a larger number of entity types. Furthermore, semantically similar fine-grained types may generate overlapping or ambiguous candidates, making candidate discrimination more challenging.

Label-wise Calibration also incurs additional computational costs as the label space expands. Specifically, its average input-token consumption increases from approximately 430 to 2,020 tokens per sentence, primarily because the calibration prompts need to accommodate more entity-type descriptions. This indicates that the calibration stage also becomes increasingly expensive as the number of entity types grows.

Overall, these additional experiments demonstrate that ProFiNER can be extended to a substantially larger label space containing 66 fine-grained entity types while maintaining stable boundary extraction and coarse-grained recognition. However, fine-grained type discrimination introduces additional challenges and computational overhead. These findings also identify type-wise candidate mining and label-wise calibration as the primary areas for future optimization, particularly when extending ProFiNER to even larger entity schemas.

Reviewer_sK9qOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ProFiNER, a progressive inference framework for zero-shot named entity recognition. It includes the following steps: 1) a dual candidate mining step that improves candidate coverage; 2) a boundary-aware calibration step that uses LLMs to calibrate candidate entity boundaries; 3) a type-aware calibration step that assigns entity label to candidate entities; 4) a conflict-aware selection step that applies candidate conflict gran to select a mutually compatible set of typed entities. The paper evaluated the proposed approach on four NER benchmarks and showed it achieved better results on 3 benchmarks compared with some existing approaches.

Critical strengths

The proposed approach improved both coverage and accuracy for zero-shot named entity recognition, and achieved good results on 4 NER benchmarks.

Critical weaknesses

The paper did not evaluate the latency of the proposed approach and compare it with other approaches. Latency can be an important factor that determines whether the approach is practical for real applications.

Questions for authors

One suggestion to improve latency is to apply the proposed approach with a strong LLM teacher to generate high-quality training data, and train a small student model to reduce latency without sacrificing accuracy.

1 comment
AuthorsOpenReview ↗

Additional Teacher–Student Experiments for Inference Efficiency

Response to W1 and Q1: Evaluation of Inference Latency and Teacher–Student Distillation

Thank you for this valuable suggestion. We agree that inference latency is important for practical applications. Following your suggestion, we have supplemented teacher–student experiments to evaluate the potential of ProFiNER for model lightweighting and inference acceleration.

1. Additional Teacher–Student Experiments.

We adopt EvoPrompt and ProFiNER, both based on the same Qwen2.5-32B backbone, as teacher approaches to generate pseudo-labeled data on identical training texts across four datasets: CoNLL03, Movie, WikiGold, and Restaurant. We then independently train BERT student models using these pseudo-labels under identical configurations and evaluate them on the corresponding test sets. Performance is measured using strict entity-level F1, requiring both entity boundaries and types to be correct. The results are shown in the following table.

DatasetEvoPrompt Teacher F1ProFiNER Teacher F1EvoPrompt→BERT F1ProFiNER→BERT F1
CoNLL0379.3183.0776.9583.16
Movie76.3876.0075.9375.90
WikiGold74.0279.0072.7574.95
Restaurant68.5672.0757.7471.16
Average74.5777.5470.8476.29

2. Comparative Analysis of Student Performance.

As shown in the table, the ProFiNER student achieves an average F1 of 76.29%, outperforming the EvoPrompt student (70.84%) by 5.45 percentage points. ProFiNER also achieves higher student F1 scores on CoNLL03, WikiGold, and Restaurant, with a particularly notable improvement of 13.42 percentage points on Restaurant. Furthermore, the ProFiNER student retains most of its teacher's performance, with an average F1 decrease of only 1.24 percentage points based on unrounded averages. These results demonstrate that ProFiNER provides more effective pseudo-label supervision and facilitates better knowledge transfer to lightweight BERT models.

3. Evaluation of Inference Latency and Practical Efficiency.

We further measure the per-sentence inference time of the teacher and student models. The ProFiNER student requires only 0.00433 seconds per sentence, compared with 1.0804 seconds for its teacher. Similarly, the EvoPrompt student requires 0.00485 seconds, compared with 1.3683 seconds for its teacher.

Specifically, the ProFiNER student achieves an approximately 249.5-fold speedup, while the EvoPrompt student achieves an approximately 282.1-fold speedup. This acceleration is achieved by generating pseudo-labels offline and deploying lightweight BERT models for subsequent inference, avoiding repeated LLM inference during practical deployment.

Overall, these experiments demonstrate that ProFiNER serves not only as an effective zero-shot NER framework but also as a high-quality teacher for lightweight model development. It enables more effective knowledge transfer than EvoPrompt while substantially reducing inference latency, offering a practical balance between extraction performance and computational efficiency.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Dear Area Chair and Reviewers,

We sincerely thank you for your time and constructive feedback. We appreciate the reviewers' valuable suggestions regarding computational efficiency, scalability, robustness, and methodological clarity. We have revised the manuscript and provided point-by-point responses, supported by additional experiments and detailed analyses.

ProFiNER presents a tuning-free named entity recognition framework that integrates dual candidate mining, boundary-aware calibration, type-aware calibration, and conflict-aware selection to improve entity extraction through complementary evidence and progressive refinement. Below, we summarize the main revisions and additional evidence supporting our contribution.

1. Controlled comparisons and computational efficiency. We conducted additional comparisons with representative tuning-free NER methods using the same Qwen2.5-32B backbone. ProFiNER achieves the highest average F1 of 77.54%, outperforming EvoPrompt by 2.97 percentage points while reducing inference latency and GPU consumption by approximately 21%. We also provided a stage-wise analysis of token consumption, inference latency, and GPU runtime to clarify the performance–efficiency trade-offs and identify the primary computational bottlenecks.

2. Inference acceleration through knowledge distillation. We introduced teacher–student experiments using ProFiNER and EvoPrompt to generate pseudo-labeled data for lightweight BERT models. The ProFiNER-trained student achieves an average F1 of 76.29%, exceeding the EvoPrompt-trained student by 5.45 percentage points. Meanwhile, its inference latency decreases to 0.00433 seconds per sentence, representing an approximately 249.5-fold speedup over the original ProFiNER framework.

3. Scalability to larger entity schemas. We extended the evaluation to Few-NERD, comparing 8 coarse-grained and 66 fine-grained entity types. The results demonstrate stable boundary extraction performance (76.66% vs. 76.49%) despite the expanded label space. We further analyzed fine-grained classification accuracy and computational overhead, clarifying both the scalability of ProFiNER and its remaining challenges.

4. Hyperparameter robustness and calibration stability. We added sensitivity analyses of scoring weights and sampling configurations. The results show that moderate variations in scoring weights lead to F1 changes within approximately 0.5 percentage points, while revised sampling configurations reduce the nominal sampling budget by 60–80% with minimal loss in candidate boundary recall. Additional mini-batch and per-sentence experiments demonstrate that type-aware calibration remains effective without relying on full-test-set statistics.

5. Boundary calibration and implementation details. We supplemented the analysis of Boundary-aware Calibration with detailed span-level tracing experiments, quantifying successful corrections, potentially harmful modifications, and their downstream effects on entity type assignments and final predictions. We also clarified the score assignment mechanism for newly corrected spans, providing a more transparent characterization of the method's behavior and limitations.

We hope these revisions and additional experiments address the concerns raised and provide stronger evidence for the effectiveness, robustness, and practical applicability of ProFiNER. We welcome any further questions and sincerely thank you again for your thoughtful consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.