Validation of an AI-powered mobile application for personalizing medical note explanations: a mixed-methods evaluation

Abstract

Introduction Nearly half of adults struggle to understand written health information, making medical communication a persistent barrier to effective care. While artificial intelligence has potential to improve health communication, few patient-facing tools have undergone systematic validation for personalized medical explanation. Methods Patiently AI is a mobile application designed to clarify clinician-authored medical notes using large language models with audience-specific adaptations (child, teenager, adult, carer) and tone variations (friendly, informative, reassuring). A three-phase mixed-methods evaluation was conducted: (1) computational readability analysis of 210 AI-generated explanations using established metrics; (2) expert review by 15 healthcare professionals assessing medical accuracy, safety, and communication quality; and (3) a patient survey of 54 participants evaluating preferences, comprehension, and acceptance. Results AI-generated explanations demonstrated consistent improvements in readability, with mean Flesch–Kincaid Grade Level decreasing by 2.96 levels (10.57–7.61), Flesch Reading Ease increasing by 31.9 points (37.7–69.6), and Gunning Fog Index decreasing by 4.09 points (14.5–10.4); all improvements were statistically significant (all P ≤ 0.002). Readability gains were greatest for younger audiences (child: 4.25 grade-level reduction; adult: 1.80). Expert reviewers rated outputs highly for medical accuracy (4.49 ± 0.83/5), clarity (4.53 ± 0.77/5), and trustworthiness (4.37 ± 0.90/5), with 87.3% assessed as clinically safe. Inter-rater agreement across the 15 reviewers was substantial (Gwet's AC1 = 0.72 for safety assessments). Among patients, 70.0% of responses preferred AI-generated explanations ( P < 0.001), with 98.1% comprehension accuracy and high ratings for clarity (4.58 ± 0.65/5) and confidence in care (4.19 ± 0.85/5). Overall, 70.4% indicated a likelihood of using the application. Conclusions This mixed-methods evaluation suggests that a deliberately constrained, language-focused AI system can improve the accessibility of medical notes while preserving clinical accuracy and safety. Patiently AI demonstrates a scalable approach to supporting health literacy and patient engagement without extending into clinical interpretation.