Avaliando modelos de linguagem de grande escala para detecção de anafilaxia em anotações clínicas
DOI:
https://doi.org/10.59681/2175-4411.v16.iEspecial.2024.1364Keywords:
Anaphylaxis, Large Language Models, Artificial IntelligenceAbstract
Objective: This study aims to evaluate the potential of four Large Language Models (LLMs) (GPT-4 Turbo, GPT-3.5 Turbo, Gemini 1.0 Pro, and OpenChat 3.5) in detecting anaphylaxis in Electronic Medical Records (EMRs). Method: The method employed involved the analysis of 150 medical reports, using different prompts to test the ability of the LLMs to identify anaphylaxis. Results: The results indicate that all models obtained zero false negatives, with GPT-4 Turbo standing out, achieving 97% accuracy and 91% precision. Conclusion: It is concluded that LLMs demonstrate the potential to assist in the identification of anaphylaxis, especially GPT-4 Turbo. The research reinforces the importance of efficient prompt design to optimize the accuracy of results.
References
Simons FER, Ardusso LR, Bilò MB, Cardona V, Ebisawa M, El-Gamal YM, et al. International consensus on (ICON) anaphylaxis. World Allergy Organ J. 2014;7(1):9. doi: 10.1186/1939-4551-7-9. PMID: 24920969; PMCID: PMC4038846. DOI: https://doi.org/10.1186/1939-4551-7-9
Ensina LF, Min TK, Félix MMR, de Alcântara CT, Costa C. Acute urticaria and anaphylaxis: Differences and similarities in clinical management. Front Allergy. 2022;3:840999. Available from: https://www.frontiersin.org/articles/10.3389/falgy.2022.840999. DOI: https://doi.org/10.3389/falgy.2022.840999
Cardona V, Ansotegui IJ, Ebisawa M, El-Gamal Y, Fernandez Rivas M, Fineman S, et al. World Allergy Organization anaphylaxis guidance 2020. World Allergy Organ J. 2020 Oct;13(10):100472. doi: 10.1016/j.waojou.2020.100472. Available from: https://linkinghub.elsevier.com/retrieve/pii/S1939455120303756. DOI: https://doi.org/10.1016/j.waojou.2020.100472
Liu X, Shi Y, Zhang D, Chen M, Xu Y, Zhao J, et al. Management of immune related adverse events through electronic multidisciplinary consultation: Five years of experience from Peking Union Medical College Hospital. J Clin Oncol. 2023;41(16_suppl)
. doi: 10.1200/JCO.2023.41.16_suppl.e14712. Available from: https://doi.org/10.1200/JCO.2023.41.16_suppl.e14712. DOI: https://doi.org/10.1200/JCO.2023.41.16_suppl.e14712
Gao Y, Li R, Caskey J, Dligach D, Miller T, Churpek MM, et al. Leveraging a medical knowledge graph into large language models for diagnosis prediction. arXiv [cs]. 2023 Aug. Available from: http://arxiv.org/abs/2308.14321. DOI: https://doi.org/10.2196/preprints.58670
Carrell DS, Gruber S, Floyd JS, Bann MA, Cushing-Haugen KL, Johnson RL, et al. Improving methods of identifying anaphylaxis for medical product safety surveillance using natural language processing and machine learning. Am J Epidemiol. 2023 Feb;192(2):283-295. doi: 10.1093/aje/kwac182. Available from: https://academic.oup.com/aje/article/192/2/283/6795959. DOI: https://doi.org/10.1093/aje/kwac182
Kural KC, Mazo I, Walderhaug M, Santana-Quintero L, Karagiannis K, Thompson EE, et al. Using machine learning to improve anaphylaxis case identification in medical claims data. JAMIA Open. 2023 Oct;6(4). doi: 10.1093/jamiaopen/ooad090. Available from: https://academic.oup.com/jamiaopen/article/doi/10.1093/jamiaopen/ooad090/7331170. DOI: https://doi.org/10.1093/jamiaopen/ooad090
Tu H, Han L, Nenadic G. Extraction of medication and temporal relation from clinical text using neural language models. arXiv [cs]. 2023 Oct. doi: 10.48550/arXiv.2310.02229. Available from: http://arxiv.org/abs/2310.02229. DOI: https://doi.org/10.1109/BigData59044.2023.10386489
Pan J, Zhang Z, Peters SR, Vatanpour S, Walker RL, Lee S, et al. Cerebrovascular disease case identification in inpatient electronic medical record data using natural language processing. Brain Inform. 2023 Sep;10(1):22. doi: 10.1186/s40708-023-00203-w. Available from: https://doi.org/10.1186/s40708-023-00203-w. DOI: https://doi.org/10.1186/s40708-023-00203-w
Lin E, Zwolinski R, Wu JT, La J, Goryachev S, Huhmann L, et al. Machine learning-based natural language processing to extract PD-L1 expression levels from clinical notes. Health Inform J. 2023 Jul;29(3):14604582231198021. doi: 10.1177/14604582231198021. Available from: http://journals.sagepub.com/doi/10.1177/14604582231198021. DOI: https://doi.org/10.1177/14604582231198021
Zitu MM, Zhang S, Owen DH, Chiang C, Li L. Generalizability of machine learning methods in detecting adverse drug events from clinical narratives in electronic medical records. Front Pharmacol. 2023 Jul;14:1218679. doi: 10.3389/fphar.2023.1218679. Available from: https://www.frontiersin.org/articles/10.3389/fphar.2023.1218679/full. DOI: https://doi.org/10.3389/fphar.2023.1218679
Afshar M, Adelaine S, Resnik F, Mundt MP, Long J, Leaf M, et al. Deployment of real-time natural language processing and deep learning clinical decision support in the electronic health record: Pipeline implementation for an opioid misuse screener in hospitalized adults. JMIR Med Inform. 2023 Apr;11. doi: 10.2196/44977. Available from: https://medinform.jmir.org/2023/1/e44977. DOI: https://doi.org/10.2196/44977
Oliveira LES, Peters AC, Silva AMP, Gebeluca CP, Gumiel YB, Cintho LMM, et al. SemClinBr - a multi-institutional and multi-specialty semantically annotated corpus for Portuguese clinical NLP tasks. J Biomed Semantics. 2022 May;13(1):1. doi: 10.1186/s13326-022-00269-1. Available from: https://doi.org/10.1186/s13326-022-00269-1. DOI: https://doi.org/10.1186/s13326-022-00269-1
Wang G, Cheng S, Zhan X, Li X, Song S, Liu Y. OpenChat: Advancing open-source language models with mixed-quality data. arXiv [cs.CL]. 2024. Available from: https://arxiv.org/abs/2309.11235.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Submission of a paper to Journal of Health Informatics is understood to imply that it is not being considered for publication elsewhere and that the author(s) permission to publish his/her (their) article(s) in this Journal implies the exclusive authorization of the publishers to deal with all issues concerning the copyright therein. Upon the submission of an article, authors will be asked to sign a Copyright Notice. Acceptance of the agreement will ensure the widest possible dissemination of information. An e-mail will be sent to the corresponding author confirming receipt of the manuscript and acceptance of the agreement.
