acceptodds
Under review as a conference paper at ICLR 2027

Separating the Model from the Decision Rule in the Evaluation of Long-Tailed ICD Coding

Abstract

Automated ICD coding is extreme multi-label classification over thousands of long-tailed codes; most reported progress since 2023 is in macro-F1 and has been credited to new model components. Macro-F1, however, also depends on the decision rule, the thresholds that turn a model's scores into codes. Published systems on the standard MIMIC-IV ICD-10 benchmarks differ in how they set these thresholds or do not state how, so the model's share of any reported gain is unknown. Changed on their own, the thresholds can add as much macro-F1 as nearly half of that progress and alter how systems compare; we therefore propose a reporting standard that separates the model from the rule. The benchmark's reference protocol tunes one threshold on validation for micro-F1; using one per training-frequency bin, tuned for twice micro-F1 plus macro-F1, instead raises the macro-F1 of PLM-ICD, the 2023 state of the art, by +0.034, 46% of the progress since. On the reproduced members of the state of the art, GoM-ICD, the same change lifts one member alone above the published three-member ensemble on macro-F1. The gain is positive on all of our more than 60 training runs and all five released checkpoints we scored. The reason is that rare codes keep a high per-label AUC while a threshold shared with common codes sits above their positives: they are rankable but not decidable. Pre-registered tests on legal and protein-function benchmarks find the same, and the gain grows with how well the tail is ranked. Reporting both rules also shows which components improve the model: averaging members that rank codes differently gains under both, while negative focusing in the loss only moves the operating point. Built from the components that gain under both, an average of three long-context clinical encoders sets the state of the art on both standard benchmarks. On the 7,942-code benchmark it improves on GoM-ICD by +0.057 macro-F1 at higher micro-F1, of which +0.048 comes from the thresholds; on the 26,096-code benchmark it reaches 1.4x the best published macro-F1 at equal micro-F1. The proposed standard reports every result under both rules; a difference that holds under both is a difference between models.

Primary area:
datasets and benchmarks
Keywords:
evaluation protocols, decision rules, threshold selection, macro-F1, long-tailed classification, extreme multi-label classification, automated ICD coding, clinical NLP, benchmarking, MIMIC-IV

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JgQEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits evaluation practices in long-tailed automated ICD coding on the MIMIC-IV ICD-10 benchmarks. It identifies widespread inconsistencies in decision-threshold setting across published systems and quantifies the impact on frozen models: replacing the standard single global threshold (tuned for micro-F1; "Rule A") with five frequency-binned thresholds (tuned for 2⋅micro-F1+macro-F12 \cdot \text{micro-F1} + \text{macro-F1}; "Rule B") yields consistent macro-F1 gains across models (+0.034 on PLM-ICD, framed as 46% of benchmark macro-F1 progress since 2023). Pre-registered experiments extend these findings to legal and protein-function domains. The authors propose dual-rule reporting and introduce a three-encoder ensemble achieving state-of-the-art results, with +0.048 of its +0.057 macro-F1 margin over GoM-ICD driven by threshold selection.

Critical strengths
  • Rigorous Evaluation Audit: Exposes a prevalent evaluation flaw across the ICD-10 literature with tight methodological controls: frozen logits, validation-only threshold fitting, bootstrap confidence intervals, open checkpoints, SOTA reproductions, and pre-registered hypotheses.
Critical weaknesses
  1. Trade-Off Shift Rather Than Representation/Ranking Improvement: Rule B is purely a post-hoc, label-wise threshold re-weighting on frozen logits. Consequently, all threshold-free ranking metrics (per-label AUC, macro-AUPRC, Precision@kk) remain entirely identical. The macro-F1 gains come at the expense of micro-F1 (averaging −0.006-0.006 on the 7.9k split and −0.028-0.028 on the 26.1k split). Tuning distinct thresholds for rare vs. frequent labels is a known property of F1-optimal thresholding (Lipton et al., 2014) and logit adjustment (Menon et al., 2021). Prior work also highlighted threshold effects (Edin et al., 2023) and frequency-grouped thresholds (Choi et al., 2026). The paper provides no algorithmic innovation that improves rare-code ranking capacity.
  2. Flawed Reporting Proposal & Overstated Headlines:
    • Incomplete Standard: If the goal is decoupling model quality from thresholding rules, threshold-free metrics (macro-AUPRC, macro-AUC, Precision@kk) or full micro-macro trade-off curves are superior to fixing two arbitrary heuristic rules (2⋅micro+macro2\cdot\text{micro} + \text{macro}).
    • Misleading Framing: Framing threshold gains as "46% of progress since 2023" conflates post-hoc operating point shifts with architectural progress. Author reproductions show genuine model improvements persist under both rules (e.g., GoM-ICD beats PLM-ICD by +0.024+0.024 under Rule A and +0.018+0.018 under Rule B).
    • Confounded SOTA Claim: SOTA gains are overwhelmingly threshold-driven (+0.048+0.048 out of +0.057+0.057). Under Rule A, the base single model falls behind GoM-ICD (0.2690.269 vs. 0.2850.285 macro-F1). Furthermore, the pre-training corpus likely contains test set notes.
Questions for authors
  1. Threshold-Free & Ranking Metrics: Can you report threshold-free ranking metrics (macro-AUPRC, macro-AUC, Precision@kk, and these restricted to rare codes) for your system, PLM-ICD, and reproduced GoM-ICD variants? Does your system actually improve rare-code ranking, or merely shift the operating point? (Demonstrating superior rare-code ranking capability would significantly strengthen the paper.)
  2. Precision-Recall Trade-off on Rare Codes: For rare codes, is Rule B's macro-F1 gain driven predominantly by recall at the cost of precision? Can you provide per-bin Precision and Recall breakdowns to assess practical utility in clinical coding workflows where false positives incur high costs?
  3. Rationale for Proposed Standard: Why advocate for two specific heuristic threshold rules rather than threshold-free metrics or full micro-macro Pareto frontiers? How sensitive are the conclusions to bin edges and objective weightings?
  4. Novelty vs. Prior Art: How do Rule B and its mechanism differ from Choi et al. (2026), beyond the number of frequency bins and hyperparameter weighting?
2 comments
AuthorsOpenReview ↗

Response to Reviewer JgQE: our system ranks rare codes better and leads GoM-ICD under the MIMIC-IV benchmark's own rule

Thank you for the detailed review. Your main point is that a change of thresholds moves the operating point and does not improve the model. We agree, and this is what the paper argues. Thresholds alone can change a model's macro-F1 by more than the margin between leading systems, so a reported gain shows model progress only when the decision rule is the same.

Rule A, the benchmark's standard protocol, can rank a weaker model above a stronger one. Among nine of our models that differ only in training settings, rule A ranks the one with the lowest macro-AP first and the one with the highest last, 0.030 macro-F1 apart. Under rule B the nine are within 0.012.

You wrote that "demonstrating superior rare-code ranking capability would significantly strengthen the paper." Our system ranks rare codes better (AP 0.323, against 0.249 for the best reproduced GoM-ICD member and 0.216 for released PLM-ICD), and so does the same recipe without clinical pretraining (0.284). The revised main table reports these numbers.

The ranking metrics you requested, on test ("rare" means 1 to 20 training examples):

macro-APrare APmacro-AUCrare AUCP@8
ours, three models0.3880.3230.97850.9790.736
ours, base model0.3700.3050.97660.9770.731
control, no clinical pretraining0.3550.2840.97560.9770.727
best GoM-ICD member, reproduced0.3190.2490.96680.9640.710
released PLM-ICD0.2950.2160.96670.9650.703

As for whether rule B's gain on rare codes comes from recall at the cost of precision, it does. Precision / recall on the test set, pooled over the codes in each bin (the revised appendix has all four bins):

training examplesPLM-ICD, rule APLM-ICD, rule Bours, rule Aours, rule B
1 to 200.38 / 0.120.20 / 0.230.51 / 0.180.28 / 0.35
over 1,0000.67 / 0.680.65 / 0.690.68 / 0.710.68 / 0.72

Under rule A, PLM-ICD doesn't assign 67% of the rare codes to any note. Where false positives cost more, a weight of 16 on micro-F1 keeps rule A's micro-F1 for our system and rare-code precision at 0.45, and still adds +0.010 macro-F1. We propose rule B for comparing models, but a deployed system should set its thresholds based on its needs.

Rule B is not only a trade-off shift. At the same micro-F1, it gives higher macro-F1 than any single threshold, even one chosen on test.

We propose reporting under two rules, the benchmark's single threshold (rule A) and our binned thresholds (rule B), because progress is reported as F1, and F1 depends on the decision rule. A ranking metric doesn't score the list of codes a system outputs, and doesn't show how much of a published F1 gain comes from the decision rule. In your example, GoM-ICD beats PLM-ICD under both rules, which shows that this gain doesn't come from the rule. We prefer two rules to a trade-off curve because two numbers fit in a results table and can be compared across papers. As you suggest, the standard also reports macro-AP (overall and on rare codes) and P@k. With other bin edges, equal-count bins, Choi et al.'s groups or other objective weights, the released and GoM-ICD models each keep their order.

Under rule A, the system leads GoM-ICD. Our Table 4 invited the comparison of 0.269 with 0.285, two different forms of macro-F1. In GoM-ICD's form, under rule A:

micro-F1 / macro-F1
GoM-ICD, published (three-member ensemble)0.613 / 0.285
ours, base model (single model)0.617 / 0.288
control, no clinical pretraining (single model)0.613 / 0.283
ours, three models0.626 / 0.315

The revision states the lead this way, and shows the +0.034 next to the 0.074 of published progress without the percentage.

You mentioned our encoder may have seen the benchmark's notes as unlabeled text, so we retrained the base model from a general-domain encoder with no clinical pretraining. This control loses only 0.004 micro-F1, and as a single model it still matches GoM-ICD's published three-member ensemble. The rule's gains do not depend on our encoder, because they also appear on released and reproduced checkpoints that could not have seen these notes.

Rule B differs little from Choi et al.'s rule, and our paper does not claim the rule as a new method. At a matched objective the two agree within about 0.005 macro-F1. Choi et al. apply group thresholds to PLM-ICD models that they trained. They do not apply them to released or published systems, do not study whether comparisons between models change or what determines the gain, and do not propose a reporting standard. Our paper does all three, using released checkpoints and reproduced GoM-ICD members.

We would be grateful if you would reconsider your score.

Dear Reviewer JgQE, as the discussion period ends on Nov 18, we wanted to check whether our response and revision address your concerns, including the rare-code ranking comparison you asked for (rare-code AP 0.323, against 0.249 for the best GoM-ICD member and 0.216 for PLM-ICD). If any concern remains, we would welcome the chance to address it before the discussion closes.

Reviewer_q5KnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how decision rules affect the evaluation of long-tailed ICD coding. It compares a global threshold optimized for validation micro-F1 (Rule A) with five frequency-group thresholds optimized for a combined micro–macro objective (Rule B). Through frozen-logit experiments, checkpoint analyses, reproduced GoM-ICD models, two MIMIC-IV benchmarks, and external label spaces, the paper shows that threshold selection can substantially change macro-F1 and the differences between models. It consequently proposes reporting results under both rules with explicit metric conventions.

The central empirical finding is well supported. My main reservations concern the incremental contribution beyond existing ICD threshold studies, the interpretation of “model–decision-rule separation,” and the comparability of the accompanying SOTA claims.

Critical strengths

S1. The paper addresses an important evaluation problem. Macro-F1 is sensitive to decision thresholds, particularly for rare labels. Quantifying this sensitivity is useful for interpreting benchmark progress and avoiding misleading attribution of gains to model components.

S2. The controlled experiments provide convincing evidence of a decision-rule effect. Keeping logits fixed isolates the consequences of changing the decision procedure. Section 3.3 and Table 1 additionally distinguish threshold granularity from the optimization objective, strengthening the analysis beyond a simple comparison of one threshold against five.

S3. The study offers useful breadth and transparent reporting. The evaluation includes multiple released checkpoints, reproduced GoM-ICD members, two ICD benchmarks, and external tasks. The paper explicitly documents metric conventions, reproduction limitations, possible pretraining overlap, and unsupported preregistered predictions. These disclosures improve the interpretability of the results.

Critical weaknesses

W1. The incremental evaluation contribution beyond closely related work needs sharper demonstration. Choi et al. (2026) already investigated global, per-label, and frequency-group thresholds for ICD coding, including separate micro-F1 and macro-F1 optimization. The manuscript appropriately acknowledges this work in Sections 2 and 7. Its contribution should therefore be assessed through the additional audit findings and reporting implications rather than the novelty of grouped thresholding itself. The checkpoint analyses, changes in model comparisons, and external evaluations provide additional evidence. However, the paper would benefit from a more explicit account of which conclusions extend beyond prior findings. Applying a representative Choi-style three-group rule to the same frozen logits would help establish whether the main conclusions and the rationale for the proposed reporting standard depend on the particular five-group rule. This comparison could use existing outputs without requiring a full retraining of the prior system.

W2. The proposed separation is operational and does not fully distinguish ranking quality from calibration. Reporting both rules establishes whether an improvement persists across two specified decision procedures. It does not generally identify a calibration-independent improvement in representations or ranking. For example, label-specific additive logit offsets preserve every label’s ranking but can alter performance under both rules because neither rule resolves calibration differences within a frequency group. The abstract’s concluding statement that a difference holding under both rules is a difference between models therefore requires a precise definition of “model.” The interpretation is reasonable if calibration is included in model output quality, but stronger attribution to representation or ranking improvements would require additional evidence. The paper already reports frequency-stratified per-label AUROC and related analyses in Sections 3.2–3.4 and Section 5. Frequency-stratified average precision would complement these results, particularly because high AUROC under extreme imbalance does not necessarily imply useful precision at relevant recall levels.

W3. The system-level results do not isolate the sources of the underlying model improvement. Table 3 provides useful comparisons for ensembles, EMA, classifier retraining, and loss settings. Nevertheless, the base-model comparison against released PLM-ICD checkpoints combines changes in pretrained encoder, pretraining data, context processing, and implementation. It supports an improvement of the overall system under the tested rules, rather than identifying the contribution of a particular architectural choice. Sections 6 and 9 also disclose that BioClinical-ModernBERT pretraining includes MIMIC-IV notes, potentially including benchmark test notes. This raises a comparability question for the SOTA and generalization claims. The actual benchmark-text overlap and its effect remain unquantified. This limitation should constrain the interpretation of system superiority while being distinguished from the frozen-logit decision-rule finding.

W4. The historical-progress and SOTA comparisons require careful qualification. Appendix E and Table 9 appropriately document different macro-F1 definitions, and these differences do not automatically favor the proposed system. However, the published GoM-ICD ensemble has not been evaluated under the same complete set of rules and conventions. Consequently, the approximately 0.048 improvement from changing thresholds on the authors’ own system should be interpreted as a within-system rule effect, rather than a complete causal decomposition of its approximately 0.057 margin over the published GoM-ICD result. Moreover, each GoM member is reproduced with one seed, and the best reproduced member remains 0.010 below its published micro-F1. A reproduction shortfall in micro-F1 alone does not establish that comparisons of macro-F1 or threshold gains are conservative. These experiments support the rule effect on the reproduced models, but extrapolation to the original ensemble warrants caution.

Questions for authors

Q1. Which principal conclusions extend beyond the threshold analyses of Choi et al.? Would those conclusions remain unchanged when applying their frequency-group rule to the same frozen logits under matched tuning objectives and evaluation conventions?

Q2. Does “model–decision-rule separation” mean robustness across the two specified rules, or an attempt to isolate calibration-independent model quality? Can the terminology and attribution claims be aligned with this distinction?

Q3. What evidence is available about overlap between the encoder’s pretraining corpus and benchmark test texts? How should the SOTA comparisons be interpreted given differences in pretraining exposure and context processing?

4 comments
AuthorsOpenReview ↗

Response to Reviewer q5Kn: reported progress on this benchmark cannot be read as model progress, under our rule or Choi et al.'s

Thank you for the careful review. You write that the central empirical finding is well supported, and that the contribution should be assessed "through the additional audit findings and reporting implications rather than the novelty of grouped thresholding itself." We agree. The paper adds three audit findings that Choi et al. do not report, and a reporting standard that follows from them, set out under W1 below. We also ran both analyses you proposed and adopted the wording you suggested.

W1 / Q1. Under Choi et al.'s rule, every conclusion holds. We applied Choi et al.'s rule to all systems we score. Micro-F1 / macro-F1:

rule A (benchmark)rule B (ours)Choi, tuned for micro-F1Choi, tuned for macro-F1Choi groups, our objective
released PLM-ICD0.589 / 0.2250.583 / 0.2590.584 / 0.2530.565 / 0.2690.585 / 0.254
our three-model system0.626 / 0.2940.620 / 0.3420.622 / 0.3310.605 / 0.3490.618 / 0.344

Their rule also raises macro-F1 on all systems, and with the objective matched (last column) it agrees with ours within about 0.005 macro-F1.

You also asked which conclusions go beyond their study. Choi et al. show that group thresholds raise the macro-F1 of a model. Our conclusions are about what this means for the results published on this benchmark:

  • Reported progress on this benchmark cannot be read as model progress. Of the six leading systems on this benchmark, two don't state their decision rule, including the state of the art, and the other four each use a different one. The rule moves macro-F1 by 0.034 on the released PLM-ICD checkpoint, while the current state of the art leads its predecessor by only 0.008.
  • The benchmark's standard protocol credits changes that do not improve the model. The standard protocol uses one threshold tuned for micro-F1. Among nine of our models that differ only in training settings, it ranks the one with the lowest macro-AP first and the one with the highest last, 0.030 macro-F1 apart. Under rule B the nine are within 0.012. An approach can therefore look effective because it shifts the operating point.
  • The effect goes beyond ICD coding. On a legal benchmark and in protein-function prediction, binned thresholds add around +0.02 macro-F1 where rare labels are ranked well.

The reporting standard follows from these findings. It lets published results on MIMIC-IV, the standard public benchmark for automated ICD-10 coding, be re-scored and new ones compared on equal terms. This matters beyond research, because ICD codes determine how hospitals are reimbursed and are the basis of national disease and mortality statistics.

W2 / Q2. "Separation" means robustness across the two rules. The revision defines "model" as the score function, calibration included, and the abstract was updated to reflect this. As you suggest, we added average precision by frequency (below is PLM-ICD):

training examplesAPF1, rule AF1, rule B
1 to 200.2160.1010.152
over 1,0000.5970.5910.595

On frequent codes, F1 under rule A is about equal to AP, while on rare codes it is less than half of AP. You wrote that "stronger attribution to representation or ranking improvements would require additional evidence." Our system ranks rare codes better. Its AP is higher than PLM-ICD's on 72% of rare codes.

W3 / Q3. Without any clinical pretraining, the base model loses only 0.004 micro-F1. The encoder's pretraining data includes MIMIC-IV notes, so we retrained the base model from a general-domain encoder with no clinical pretraining. This control, a single model, still matches GoM-ICD's published three-member ensemble. The decision-rule findings do not depend on our encoder, because they also appear on released and reproduced checkpoints that could not have seen these notes. Our revision reports this new control.

W4. Under the benchmark's rule, the system leads GoM-ICD by 0.030 macro-F1. As you said, the +0.048 is what binned thresholds add to our system, and is not a share of the margin over GoM-ICD. Under rule A, in the form of macro-F1 that GoM-ICD reports, our system scores 0.626 / 0.315 against the published 0.613 / 0.285. As per your suggestion, the revision no longer calls the GoM-ICD comparison conservative, claims the rule effect only for the reproduced GoM-ICD members, and gives the +0.034 next to the 0.074 of published progress.

We would be grateful if you would reconsider your score.

Dear Reviewer q5Kn, as the discussion period ends on Nov 18, we wanted to check whether our response and revision address your concerns, including Choi et al.'s rule on the same frozen scores and your suggested average precision by frequency. If any concern remains, we would welcome the chance to address it before the discussion closes.

Thank you for the detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns. The comparison with Choi et al., the additional AP analysis, and the general-domain encoder experiment provide useful evidence and improve the clarity and credibility of the paper.

However, my main concerns remain partially unresolved. First, although the paper is now more clearly positioned as an evaluation audit rather than a new thresholding method, the incremental contribution beyond existing work is still not fully established. Second, the clarification of model–decision-rule separation is helpful, but the AP results, particularly the reported improvement on 72% of rare codes, do not sufficiently quantify the overall ranking improvement. Third, the additional pretraining control alleviates some concerns but does not fully resolve the comparability and attribution issues in the system-level results.

Overall, I acknowledge the improvements made in the rebuttal and agree that the central empirical findings are valuable. Nevertheless, I do not think the additional evidence fully addresses my concerns regarding the novelty and significance of the contribution.

Thank you for the close reading and the specific follow-up. We answer your three points in order.

1. What the paper adds beyond existing work. Choi et al. show that group thresholds raise a model's macro-F1. Three findings are not in their study or, to our knowledge, in any prior work. Of the six leading systems, two do not state their decision rule and the other four each use a different one. The rule's effect on a released checkpoint (0.034 macro-F1) is about four times the margin by which the state of the art leads its predecessor (0.008). Among nine of our models that differ only in training settings, the benchmark's own protocol ranks the one with the lowest macro-AP first and the one with the highest last. Together these mean that macro-F1 gains published on this benchmark cannot be compared as reported. Audits of this kind have been published at ICLR and NeurIPS (Gulrajani & Lopez-Paz, ICLR 2021; Agarwal et al., NeurIPS 2021). We understand that how much weight this carries is a judgment for you and the area chair.

2. The overall ranking improvement. You are right that a per-code win rate does not quantify it. The overall figures are in our response to Reviewer JgQE and in the revised main table, and we repeat them here (test set; "rare" means 1 to 20 training examples).

macro-APrare-code APmacro-AUCP@8
ours, three models0.3880.3230.97850.736
ours, base model0.3700.3050.97660.731
control, no clinical pretraining0.3550.2840.97560.727
best GoM-ICD member, reproduced0.3190.2490.96680.710
released PLM-ICD0.2950.2160.96670.703

Comparing single models, our base model leads the best reproduced GoM-ICD member by 0.051 macro-AP and 0.056 rare-code AP, and released PLM-ICD by 0.075 and 0.089. A label's AP depends only on how its scores rank the notes, so no threshold or label-specific offset, as in your W2 example, can change it. These gains are therefore in ranking, independent of the decision rule and of calibration.

3. Comparability and attribution. We agree, and we read the system-level results as you do. They show that the overall system improves, and they do not attribute the gain to the pretrained encoder, context processing or implementation separately. The control removes one confound only: without any clinical pretraining, this control, a single model, still leads the best reproduced GoM-ICD member by 0.036 macro-AP. The paper's main findings do not rest on this comparison, because the decision-rule results are measured on frozen released and reproduced checkpoints.

Thank you again for a careful and constructive review. If the figures under point 2 change your assessment, we would be grateful if you would reflect that in your final evaluation.

Reviewer_UYzrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds that for (extreme, long-tailed) multi-label classification where the decision rule for whether to assign a label to a note is usually predicted probability > threshold, if one were to optimise such thresholds per-label (or per label-group) rather than using a global threshold, Macro-F1 scores will improve. This is because the model is less confident on rare labels (seeing them less often than frequent ones), thus assigning lower probabilities that don't surpass a global threshold optimised dominated by frequent labels.

The paper shows that Macro-F1 scores improve using thresholds optimised separately for bins of labels, based on their frequency.

Critical strengths

The paper sets the record straight for MIMIC IV evaluation, suggesting a new protocol, which could be seen as valuable.

Critical weaknesses

I don't think the scale of the contribution is commensurate with the venue. Basically, the paper says that if we optimise the decision rule for whether to assign a label to a note (usually predicted probability > threshold) per-label (or per label-group) rather than using a global decision rule, this benefits Macro-F1 scores as it scores rare labels better, which, by virtue of being rare, the model is less confident on.

This isn't particularly new, neither is it specific to the MIMIC-IV data chosen as a study subject here. Yes, technically this improves the (Macro F1) score by a few points, but this phenomenon by itself is not novel enough to have a whole paper written about it.

Here's a cross-selection of papers that do similar things already (deliberately or as drive-by), some are even included as references in the paper:

Questions for authors

The manuscript mentions pre-registered tests at least twice, but i am not aware of a mechanism to pre-register experiments with ICLR. Would the authors care to elaborate?

2 comments
AuthorsOpenReview ↗

Response to Reviewer UYzr: an audit of published results, not a new thresholding method

Thank you for the review and the references. Our revision cites them all. You write that tuning thresholds per label or per group "isn't particularly new." The paper does not claim it to be. The referenced works show that such thresholds raise the macro-F1 of a model, while our conclusions are about what this means for the results published on this benchmark:

  • Reported progress on this benchmark cannot be read as model progress. You write that the rule moves macro-F1 "by a few points." On this benchmark that is far more than the margin between leading systems. The rule moves macro-F1 by 0.034 on the released PLM-ICD checkpoint, while the current state of the art leads its predecessor by only 0.008. Of the six leading systems on this benchmark, two don't state their decision rule, including the state of the art, and the other four each use a different one.
  • We show the benchmark's standard protocol credits changes that do not improve the model. The standard protocol uses one threshold tuned for micro-F1. Among nine of our models that differ only in training settings, it ranks the one with the lowest macro-AP first and the one with the highest last. A component can therefore look effective in an ablation because it shifts the operating point.
  • As you said, the effect goes beyond ICD coding. On a legal benchmark and in protein-function prediction, binned thresholds add around +0.02 macro-F1 where rare labels are ranked well.

The reporting standard follows from these findings. It lets published results on MIMIC-IV, the standard public benchmark for automated ICD-10 coding, be re-scored and new ones compared on equal terms. This matters beyond research, because ICD codes determine how hospitals are reimbursed and are the basis of national disease and mortality statistics.

ICLR and NeurIPS have published this kind of contribution before. Gulrajani & Lopez-Paz (ICLR 2021) re-evaluated domain-generalization methods under common model-selection rules and asked that every method state its rule. Agarwal et al. (NeurIPS 2021, outstanding paper award) showed that conclusions on the Atari 100k benchmark change once uncertainty is reported, and proposed a reporting standard. Our paper does the same for decision rules in ICD coding.

"Pre-registered" is corrected in the revision. We didn't use an external registry. The predictions were written in our version control before the data they concern were scored, and the dated documents are in our supplementary material.

We would be grateful if you would reconsider your score.

Dear Reviewer UYzr, as the discussion period ends on Nov 18, we wanted to check whether our response and revision address your concerns, including the clarified scope of the contribution and the correction of "pre-registered". If any concern remains, we would welcome the chance to address it before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of responses and revisions

We thank the reviewers. Their shared question is what the paper adds, since thresholds tuned per label or per group are known to raise macro-F1. The paper does not claim them as a new method. As Reviewer q5Kn writes, the contribution should be assessed "through the additional audit findings and reporting implications rather than the novelty of grouped thresholding itself." Prior work, including Choi et al. (2026), shows that such thresholds raise the macro-F1 of a model. Our conclusions are about what this means for the results published on this benchmark, and we clarified this in our revision:

  • Reported progress on this benchmark cannot be read as model progress. Of the six leading systems on this benchmark, two don't state their decision rule, including the state of the art, and the other four each use a different one. The rule moves macro-F1 by 0.034 on the released PLM-ICD checkpoint, while the current state of the art leads its predecessor by only 0.008.
  • The benchmark's standard protocol credits changes that do not improve the model. The standard protocol (rule A) uses one threshold tuned for micro-F1. Among nine of our models that differ only in training settings, it ranks the one with the lowest macro-AP first and the one with the highest last, 0.030 macro-F1 apart. Under our binned thresholds (rule B) the nine are within 0.012. An approach can therefore look effective because it shifts the operating point.
  • The effect goes beyond ICD coding. On a legal benchmark and in protein-function prediction, binned thresholds add around +0.02 macro-F1 where rare labels are ranked well.

The reporting standard follows from these findings. Every result is reported under both rules, next to macro-AP and P@k. The standard lets published results on MIMIC-IV, the standard public benchmark for automated ICD-10 coding, be re-scored and new ones compared on equal terms. This matters beyond research, because ICD codes determine how hospitals are reimbursed and are the basis of national disease and mortality statistics.

ICLR and NeurIPS have published this kind of contribution before, for example Gulrajani & Lopez-Paz (ICLR 2021) in domain generalization and Agarwal et al. (NeurIPS 2021) in deep reinforcement learning.

The requested analyses are done, and no result of the submission changes. They cover released checkpoints, reproduced GoM-ICD members and our own models, plus a retrained control.

requestresult
superior rare-code ranking, which "would significantly strengthen the paper" (JgQE)our system ranks rare codes better (AP 0.323, against 0.249 for the best reproduced GoM-ICD member and 0.216 for released PLM-ICD)
is rule B only a trade-off shift? (JgQE)no: at the same micro-F1, it gives higher macro-F1 than any single threshold
does the lead over GoM-ICD hold under the benchmark's rule? (JgQE, q5Kn)yes: in GoM-ICD's form of macro-F1, ours is 0.626 / 0.315 against the published 0.613 / 0.285
Choi et al.'s rule on the same frozen scores (q5Kn, JgQE)every conclusion holds, and with the objective matched it agrees with ours within about 0.005 macro-F1
pretraining on MIMIC-IV notes (q5Kn, JgQE)without any clinical pretraining, the base model loses only 0.004 micro-F1

The revised PDF makes the changes the reviewers asked for. It defines "model" as the score function, reports the +0.034 next to the 0.074 of published progress without a percentage, adds the threshold-free metrics and the control to the main table, cites every work the reviewers list, and corrects "pre-registered".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.