- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper audits evaluation practices in long-tailed automated ICD coding on the MIMIC-IV ICD-10 benchmarks. It identifies widespread inconsistencies in decision-threshold setting across published systems and quantifies the impact on frozen models: replacing the standard single global threshold (tuned for micro-F1; "Rule A") with five frequency-binned thresholds (tuned for ; "Rule B") yields consistent macro-F1 gains across models (+0.034 on PLM-ICD, framed as 46% of benchmark macro-F1 progress since 2023). Pre-registered experiments extend these findings to legal and protein-function domains. The authors propose dual-rule reporting and introduce a three-encoder ensemble achieving state-of-the-art results, with +0.048 of its +0.057 macro-F1 margin over GoM-ICD driven by threshold selection.
Critical strengths
- Rigorous Evaluation Audit: Exposes a prevalent evaluation flaw across the ICD-10 literature with tight methodological controls: frozen logits, validation-only threshold fitting, bootstrap confidence intervals, open checkpoints, SOTA reproductions, and pre-registered hypotheses.
Critical weaknesses
- Trade-Off Shift Rather Than Representation/Ranking Improvement: Rule B is purely a post-hoc, label-wise threshold re-weighting on frozen logits. Consequently, all threshold-free ranking metrics (per-label AUC, macro-AUPRC, Precision@) remain entirely identical. The macro-F1 gains come at the expense of micro-F1 (averaging on the 7.9k split and on the 26.1k split). Tuning distinct thresholds for rare vs. frequent labels is a known property of F1-optimal thresholding (Lipton et al., 2014) and logit adjustment (Menon et al., 2021). Prior work also highlighted threshold effects (Edin et al., 2023) and frequency-grouped thresholds (Choi et al., 2026). The paper provides no algorithmic innovation that improves rare-code ranking capacity.
- Flawed Reporting Proposal & Overstated Headlines:
- Incomplete Standard: If the goal is decoupling model quality from thresholding rules, threshold-free metrics (macro-AUPRC, macro-AUC, Precision@) or full micro-macro trade-off curves are superior to fixing two arbitrary heuristic rules ().
- Misleading Framing: Framing threshold gains as "46% of progress since 2023" conflates post-hoc operating point shifts with architectural progress. Author reproductions show genuine model improvements persist under both rules (e.g., GoM-ICD beats PLM-ICD by under Rule A and under Rule B).
- Confounded SOTA Claim: SOTA gains are overwhelmingly threshold-driven ( out of ). Under Rule A, the base single model falls behind GoM-ICD ( vs. macro-F1). Furthermore, the pre-training corpus likely contains test set notes.
Questions for authors
- Threshold-Free & Ranking Metrics: Can you report threshold-free ranking metrics (macro-AUPRC, macro-AUC, Precision@, and these restricted to rare codes) for your system, PLM-ICD, and reproduced GoM-ICD variants? Does your system actually improve rare-code ranking, or merely shift the operating point? (Demonstrating superior rare-code ranking capability would significantly strengthen the paper.)
- Precision-Recall Trade-off on Rare Codes: For rare codes, is Rule B's macro-F1 gain driven predominantly by recall at the cost of precision? Can you provide per-bin Precision and Recall breakdowns to assess practical utility in clinical coding workflows where false positives incur high costs?
- Rationale for Proposed Standard: Why advocate for two specific heuristic threshold rules rather than threshold-free metrics or full micro-macro Pareto frontiers? How sensitive are the conclusions to bin edges and objective weightings?
- Novelty vs. Prior Art: How do Rule B and its mechanism differ from Choi et al. (2026), beyond the number of frequency bins and hyperparameter weighting?
2 comments
Dear Reviewer JgQE, as the discussion period ends on Nov 18, we wanted to check whether our response and revision address your concerns, including the rare-code ranking comparison you asked for (rare-code AP 0.323, against 0.249 for the best GoM-ICD member and 0.216 for PLM-ICD). If any concern remains, we would welcome the chance to address it before the discussion closes.
Response to Reviewer JgQE: our system ranks rare codes better and leads GoM-ICD under the MIMIC-IV benchmark's own rule
Thank you for the detailed review. Your main point is that a change of thresholds moves the operating point and does not improve the model. We agree, and this is what the paper argues. Thresholds alone can change a model's macro-F1 by more than the margin between leading systems, so a reported gain shows model progress only when the decision rule is the same.
Rule A, the benchmark's standard protocol, can rank a weaker model above a stronger one. Among nine of our models that differ only in training settings, rule A ranks the one with the lowest macro-AP first and the one with the highest last, 0.030 macro-F1 apart. Under rule B the nine are within 0.012.
You wrote that "demonstrating superior rare-code ranking capability would significantly strengthen the paper." Our system ranks rare codes better (AP 0.323, against 0.249 for the best reproduced GoM-ICD member and 0.216 for released PLM-ICD), and so does the same recipe without clinical pretraining (0.284). The revised main table reports these numbers.
The ranking metrics you requested, on test ("rare" means 1 to 20 training examples):
As for whether rule B's gain on rare codes comes from recall at the cost of precision, it does. Precision / recall on the test set, pooled over the codes in each bin (the revised appendix has all four bins):
Under rule A, PLM-ICD doesn't assign 67% of the rare codes to any note. Where false positives cost more, a weight of 16 on micro-F1 keeps rule A's micro-F1 for our system and rare-code precision at 0.45, and still adds +0.010 macro-F1. We propose rule B for comparing models, but a deployed system should set its thresholds based on its needs.
Rule B is not only a trade-off shift. At the same micro-F1, it gives higher macro-F1 than any single threshold, even one chosen on test.
We propose reporting under two rules, the benchmark's single threshold (rule A) and our binned thresholds (rule B), because progress is reported as F1, and F1 depends on the decision rule. A ranking metric doesn't score the list of codes a system outputs, and doesn't show how much of a published F1 gain comes from the decision rule. In your example, GoM-ICD beats PLM-ICD under both rules, which shows that this gain doesn't come from the rule. We prefer two rules to a trade-off curve because two numbers fit in a results table and can be compared across papers. As you suggest, the standard also reports macro-AP (overall and on rare codes) and P@k. With other bin edges, equal-count bins, Choi et al.'s groups or other objective weights, the released and GoM-ICD models each keep their order.
Under rule A, the system leads GoM-ICD. Our Table 4 invited the comparison of 0.269 with 0.285, two different forms of macro-F1. In GoM-ICD's form, under rule A:
The revision states the lead this way, and shows the +0.034 next to the 0.074 of published progress without the percentage.
You mentioned our encoder may have seen the benchmark's notes as unlabeled text, so we retrained the base model from a general-domain encoder with no clinical pretraining. This control loses only 0.004 micro-F1, and as a single model it still matches GoM-ICD's published three-member ensemble. The rule's gains do not depend on our encoder, because they also appear on released and reproduced checkpoints that could not have seen these notes.
Rule B differs little from Choi et al.'s rule, and our paper does not claim the rule as a new method. At a matched objective the two agree within about 0.005 macro-F1. Choi et al. apply group thresholds to PLM-ICD models that they trained. They do not apply them to released or published systems, do not study whether comparisons between models change or what determines the gain, and do not propose a reporting standard. Our paper does all three, using released checkpoints and reproduced GoM-ICD members.
We would be grateful if you would reconsider your score.