Diagnostic Loss: Asymmetric Supervision for Radiology Report Generation

Automated radiology report generation requires models to do more than produce fluent clinical narratives: they must explicitly state supported diagnoses while avoiding unsupported ones. However, existing medical vision-language models (MedVLMs) are typically trained with token-level maximum likelihood objectives, which prioritize agreement with reference text but do not directly constrain the probability assigned to diagnostically critical terms. As a result, models can generate plausible reports that omit key diagnoses, hedge on clinically evident findings, or introduce unsupported differential diagnoses. We identify this as an objective misalignment between conventional language modeling and the asymmetric diagnostic requirements of radiology reporting. To address this problem, we introduce Diagnostic Loss (DL), a differentiable, architecture-agnostic objective that operates directly on decoder token probabilities. DL separately aggregates probability mass over supported and unsupported diagnostic vocabularies using asymmetric temporal pooling: it encourages supported diagnoses to receive sufficient probability at at least one position while suppressing spurious probability spikes for unsupported diagnoses throughout the generated report. The loss can be applied to existing 2D and 3D MedVLMs without modifying their vision backbones, language decoders, or inference procedures, and introduces no additional inference-time computation. We evaluate DL across multiple 2D and 3D MedVLM architectures on both in-distribution and out-of-distribution report-generation benchmarks. Comprehensive experiments show that DL improves diagnostic faithfulness while maintaining competitive performance on conventional language-generation metrics, demonstrating the value of aligning training objectives with the clinical semantics of report generation.