Beyond Average Risk: Diagnosis-Priority Robust Learning for Structured Radiology Report Generation

Structured generation models are commonly optimized for average performance, although low frequency semantic events may carry disproportionate importance in high stakes domains. We study this problem in structured radiology report generation, where strong aggregate text and semantic metrics can coexist with degraded reliability on rare diagnostic findings. We formulate this failure as a semantic tail robustness problem and introduce diagnosis priority robust learning, which explicitly increases optimization pressure on diagnostically important long tail semantics and underperforming groups. Our framework uses reference side semantic representations to characterize abnormality and diagnostic rarity while separating average generation quality from worst group and rare-finding reliability. Across single-source and cross-source radiology report generation settings, we find that models with similar aggregate quality can exhibit substantially different semantic-tail behavior, and that robust training shifts the operating point between average fidelity and diagnosis priority reliability. Cross source experiments further indicate that this robustness problem becomes especially relevant under heterogeneous data distributions. We evaluate the approach using standard generation metrics together with semantic, worst group, rare recall, and false negative measures. These results motivate structured generation objectives that explicitly account for rare diagnostic semantics rather than relying solely on average sequence level performance.