Diagnosis classification in EMR data using latent representations and SNOMED-CT mapping for improved medical data integration.

Abstract

Integrating medical data across hospitals has become a critical challenge in medical informatics, largely due to the heterogeneity of electronic medical record (EMR) systems. This study aims to address this issue by developing a diagnosis classification model that automatically maps diagnosis spans in EMR data to the standardized clinical ontology Systematized Nomenclature of Medicine-Clinical Terms (SNOMED-CT). We trained an embedding-based model, ClinicalBERT, on EMR data to obtain latent representations of diagnosis spans. These representations were aligned with SNOMED-CT fully specified names (FSNs) through mean squared error (MSE)-based fine-tuning and subsequently used to construct a downstream classification model. In addition, we analyzed the latent representations to investigate semantic alignment and structural characteristics in the latent space. The proposed model achieved an accuracy of 0.934, a weighted F1-score of 0.923, and a macro-averaged F1-score of 0.823 on 273 SNOMED-CT classes. Despite improved semantic alignment, the fine-tuned model demonstrated performance comparable to the base ClinicalBERT model, while remaining competitive with state-of-the-art approaches such as SapBERT (accuracy: 0.944) and BioSyn (accuracy: 0.931). Representation analysis further revealed improved clustering coherence, as evidenced by reduced similarity distances and more distinct class separation. This approach demonstrates strong potential for scalable and privacy-preserving medical concept normalization in real-world clinical environments. Fine-tuning improved semantic alignment, reducing the average Manhattan and cosine distances by 36.2% and 53.1%, respectively. Moreover, the integration of classification and representation analysis provides insights into the trade-off between semantic alignment and discriminative performance.