- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ProFiNER, a progressive framework for zero-shot NER that combines candidate mining, boundary calibration, type calibration, and conflict-aware selection. Experiments across multiple NER benchmarks and LLM backbones demonstrate consistent improvements over direct prompting, supported by component ablation studies.
Critical strengths
The method achieves consistent improvements across multiple benchmarks and LLM backbones. The ablation studies further demonstrate the effectiveness of its major components.
Critical weaknesses
-
The proposed framework introduces substantial inference overhead, requiring 5.58–13.82 model requests per sentence, as reported in Table 6. However, the paper lacks comparisons with computationally matched baselines and does not report inference latency or token costs. This makes it difficult to assess whether the performance gains justify the additional computational expense.
-
The type calibration mechanism relies on statistics computed over the entire candidate pool (Equations 6–7). This introduces dependencies across evaluation samples, potentially limiting applicability to independent or streaming inference.
Questions for authors
-
Can the authors provide comparisons with runtime cost?
-
How does the type calibration mechanism perform when statistics are computed from smaller batches or an independent unlabeled calibration set?
-
How are the hyperparameters in Appendix A selected?
5 comments
Response to W2 and Q2: Robustness of Type-aware Calibration under Different Statistical Estimation Scopes
Thank you for raising this important concern. We agree that investigating the dependence of Type-aware Calibration on the entire test-set candidate pool is important for assessing its applicability to independent and streaming inference. To address this concern, we conduct additional experiments using different statistical estimation scopes, including the full test-set candidate pool, mini-batches of different sizes (B=8, 32, and 128), and per-sentence estimation.
(1) Mini-batch Statistical Estimation.
In ProFiNER, Type-aware Calibration employs Per-Type Score Standardization to account for differences in score distributions across entity types, facilitating subsequent pairwise adjudication. To investigate whether this process depends on global statistics, we vary the scope of candidate scores used to calculate per-type means and standard deviations.
The experimental results are summarized below.
| Calibration Setting | CoNLL03 | Δ | Movie | Δ | Restaurant | Δ | WikiGold | Δ |
|---|---|---|---|---|---|---|---|---|
| Full test set | 82.53 | – | 75.45 | – | 72.24 | – | 79.00 | – |
| Mini-batch (B=8) | 81.99 | −0.54 | 75.15 | −0.30 | 72.26 | +0.02 | 78.99 | −0.01 |
| Mini-batch (B=32) | 82.04 | −0.49 | 75.19 | −0.26 | 72.20 | −0.04 | 78.99 | −0.01 |
| Mini-batch (B=128) | 82.39 | −0.14 | 75.29 | −0.16 | 72.24 | 0.00 | 78.99 | −0.01 |
| Per-sentence | 82.43 | −0.10 | 75.71 | +0.26 | 72.28 | +0.04 | 79.11 | +0.11 |
The results indicate that Type-aware Calibration maintains stable performance under different statistical estimation scopes. With B=8, F1 decreases by only 0.54, 0.30, and 0.01 percentage points on CoNLL03, Movie, and WikiGold, respectively, while Restaurant shows a slight improvement of 0.02 percentage points. As the batch size increases to 128, performance becomes even closer to full-test-set estimation, with changes of −0.14, −0.16, 0.00, and −0.01 percentage points across the four datasets. These results suggest that estimating per-type statistics from smaller candidate pools does not substantially affect extraction performance.
(2) Per-sentence Statistical Estimation.
We further investigate a per-sentence setting to determine whether Type-aware Calibration can operate independently without relying on statistics from other evaluation samples. For each sentence, we independently calculate the mean and standard deviation of each entity type using only the raw candidate scores within that sentence. The resulting scores are then standardized and passed to the same pairwise type adjudication and conflict-aware selection procedures. If an entity type has only one candidate within a sentence, its standardized score is set to zero.
For a controlled comparison on CoNLL03, we fix the 7,093 second-stage candidates generated in the original experiment and reuse their raw type scores, changing only the scope of statistical estimation. The F1 score decreases marginally from 82.5346% under full-test-set estimation to 82.4276% under per-sentence estimation, corresponding to a difference of only 0.1070 percentage points. This demonstrates that local statistical estimation can achieve performance highly comparable to global estimation.
Overall, these findings demonstrate that Per-Type Score Standardization serves as a supporting mechanism for Type-aware Calibration, while global statistics are not essential to maintaining its extraction performance. The comparable results under mini-batch and per-sentence settings indicate that ProFiNER can accommodate different statistical estimation scopes without requiring access to the entire test-set candidate pool. This provides empirical evidence of its applicability to batch-based and sentence-level inference scenarios.
Response to Q3: Hyperparameter Selection and Sensitivity Analysis
Thank you for this constructive suggestion. We clarify our hyperparameter selection strategy and provide additional sensitivity analyses of the scoring weights and sampling parameters to evaluate the robustness of ProFiNER.
1. Hyperparameter Selection Strategy.
The hyperparameters are selected based on preliminary experimental observations and the functional roles of individual components, aiming to balance candidate coverage, calibration reliability, and computational efficiency.
The original manuscript uniformly set K₁=K₂=10 across all datasets. As this conservative configuration may introduce unnecessary inference overhead, we revised the sampling parameters based on dataset-specific sensitivity results.
For the final scoring function, we retain consistent weighting coefficients across all four datasets for five evidence sources: cluster support (), boundary evidence (), initial type confidence (), calibrated type evidence (), and adjudication margin ().
| Weight | Value |
|---|---|
| 0.175 | |
| 0.500 | |
| 0.175 | |
| 0.105 | |
| 0.505 |
The revised dataset-specific sampling parameters and calibration thresholds are:
| Dataset | K₁ | K₂ | |||
|---|---|---|---|---|---|
| CoNLL03 | 3 | 1 | 0.30 | 0.40 | 0.00 |
| Movie | 3 | 5 | 0.00 | 0.50 | 0.00 |
| WikiGold | 5 | 3 | 0.57 | 0.50 | 0.50 |
| Restaurant | 3 | 2 | 0.40 | 0.35 | 0.40 |
Scoring weights are shared across datasets, whereas sampling parameters and calibration thresholds are adapted to dataset characteristics.
2. Sensitivity Analysis.
The corresponding sensitivity-analysis figures are provided in Appendix F of the revised manuscript.
(a) Sensitivity Analysis of Scoring Weights.
We conduct one-at-a-time sensitivity analyses by varying each of the five weighting coefficients while keeping all other coefficients and evidence inputs fixed. Across the evaluated parameter ranges, final entity-level F1 varies by no more than approximately 0.5 percentage points on all four datasets. This indicates that ProFiNER is relatively insensitive to moderate changes in scoring weights, supporting the robustness of its evidence integration mechanism.
(b) Sensitivity Analysis of Sampling Hyperparameters.
We vary K₁ and K₂ from 1 to 10 and evaluate their effects on initial candidate boundary recall across the four datasets. To isolate the effects of sampling configurations, all experiments use a uniform boundary threshold of =0.
As shown in Appendix F, CoNLL03 and WikiGold exhibit relatively stable recall across different sampling configurations. Movie benefits more noticeably from increasing K₂, while Restaurant shows a more moderate improvement. These results indicate that the optimal sampling budget varies across datasets.
Based on these observations, we replace the original K₁=K₂=10 configuration with the dataset-specific settings presented above. Under the same θ_b=0 evaluation, candidate boundary recall decreases by only 0.5, 0.9, 0.4, and 0.8 percentage points on CoNLL03, Movie, WikiGold, and Restaurant, respectively.
Meanwhile, K₁+K₂ decreases from 20 to 4, 8, 8, and 5, corresponding to reductions of 80%, 60%, 60%, and 75%.
Overall, final entity-level F1 remains stable under moderate variations in scoring weights, while the revised dataset-specific sampling configurations maintain candidate boundary recall close to that of the original setting. These findings support our hyperparameter selection strategy and demonstrate the robustness of ProFiNER.
Response to W1 and Q1: Runtime Cost Comparison and Stage-wise Computational Cost Analysis
Thank you for this valuable comment. We agree that runtime cost is important for evaluating the practical applicability of ProFiNER. We therefore provide same-backbone comparisons with existing tuning-free NER methods and a stage-wise computational cost analysis.
(1) Runtime Comparison with Existing Tuning-free NER Methods
We evaluate all methods using the same Qwen2.5-32B backbone, including Direct Prompting, Self-Consistency (K=5 and K=10), Self-Improving, and EvoPrompt.
The results are summarized below.
ProFiNER achieves the highest average F1 (77.54%). Compared with Self-Improving, it improves F1 by 6.93 percentage points (77.54% vs. 70.61%), while requiring slightly less inference time (1.0804 vs. 1.1020 s/sentence) and GPU consumption (4.6732 vs. 4.7668 GPU-hours), despite higher token consumption.
Compared with EvoPrompt, ProFiNER improves F1 by 2.97 percentage points and reduces input tokens, inference latency, and GPU consumption by approximately 38.9%, 21.0%, and 21.0%, respectively. Although its output-token consumption is higher (481.06 vs. 77.13), it achieves better extraction performance with lower overall measured computational costs.
Compared with Self-Consistency (K=10), ProFiNER incurs higher computational costs (1.0804 vs. 0.7067 s/sentence; 4.6732 vs. 3.0569 GPU-hours), but improves F1 by 5.40 percentage points. This illustrates the trade-off between computational overhead and extraction performance.
(2) Stage-wise Computational Cost Analysis
We further analyze the computational costs of ProFiNER's four stages. The results are presented below.
Dual Candidate Mining contributes the largest computational cost (64.64%, 3.0206 GPU-hours), followed by Type-aware Calibration (23.77%, 1.1107 GPU-hours). Boundary-aware Calibration and Conflict-aware Selection account for 8.17% and 3.42%, respectively. Together, the first two stages account for 88.41% of the total computational cost, identifying the primary optimization targets.
The complete framework consumes 2,739.61 input tokens and 481.06 output tokens per sentence, with an average latency of 1.0804 seconds and a total cost of 4.6732 GPU-hours.
Overall, same-backbone comparisons demonstrate that ProFiNER achieves the highest extraction performance among the evaluated methods while maintaining competitive efficiency relative to computationally intensive baselines. The stage-wise analysis further identifies the main computational bottlenecks and directions for future optimization