Chatbots in identification of breastfeeding issues: performance evaluation
DOI:
https://doi.org/10.59681/2175-4411.v16.iEspecial.2024.1370Keywords:
Breastfeeding, Artificial Intelligence, Expert SystemsAbstract
Objective: This study aimed to evaluate the performance of artificial intelligence-based chatbots in identifying breastfeeding-related problems. Method: The study assessed OpenAI ChatGPT-3.5, Microsoft Copilot, Google Gemini, and Lhia in identifying breastfeeding issues. Lhia chatbot is being developed by our team of researchers. Through consensus among healthcare professionals specializing in breastfeeding, a dataset of annotated main clinical complaint reports from medical records at the University Hospital of the Federal University of Maranhão was created for testing with three zero-shot prompt approaches. Results: The best performance was achieved by ChatGPT-3.5, which demonstrated accuracy ranging from 79% to 93%, fallback from 0% to 7%, and F1-score from 75% to 100%. Conclusion: Artificial intelligence-based chatbots can be a promising tool to assist mothers and healthcare professionals in the early detection of breastfeeding issues.
References
Victora CG, Bahl R, Barros AJ, França GV, Horton S, Krasevec J, et al. Breastfeeding in the 21st century: epidemiology, mechanisms, and lifelong effect. The Lancet. 2016;387(10017):475-90.
Rollins NC, Bhandari N, Hajeebhoy N, Horton S, Lutter CK, Martines JC, et al. Breastfeeding in the 21st century: epidemiology, mechanisms, and lifelong effect. Lancet. 2016;387(10017):491-504
World Health Organization. Infant and Young Child Feeding: Model Chapter for Textbooks for Medical Students and Allied Health Professionals; Technical Report; WHO: Geneva, Switzerland, 2009
Bhattacharjee NV, Schaeffer LE, Hay SI, Lu D, Schipp MF, Lazzar-Atwood A, et al. Mapping inequalities in exclusive breastfeeding in low- and middle-income countries 2000–2018. Nat Hum Behav. 2021; 5,1027–1045.
Softić A, Husić JB, Softić A, Baraković S. Health chatbot: design, implementation, acceptance and usage motivation. In 20th International Symposium Infoteh-Jahorina; 2021 17-19 March; East Sarajevo, Bosnia and Herzegovina. IEEE; 2021, pp. 1-6, Available from: doi:10.1109/INFOTEH51037.2021.9400693.
Prasad VA, Ranjith R. Intelligent chatbot for lab security and automation. In 11th international conference on computing, communication and networking Technologies, 2020 1-3 July; Kharagpur, India. IEEE, 2020, pp. 1-4, Available from: doi:10.1109/ICCCNT49239.2020.9225641.
Yadav D, Malik P, Dabas K, Singh P. Feedpal: Understanding Opportunities for Chatbots in Breastfeeding Education of Women in India. Proceedings of the ACM on Human-Computer Interaction. 2019;3(CSCW):170:1–30.
Gupta V, Arora N, Jain Y, Mokashi S, Panda C. Assessment on Adoption Behavior of First-time Mothers on the Usage of Chatbots for Breastfeeding Consultation. J Mahatma Gandhi Univ Med Sci Tech. 2021;6(2):64-68.
Montenegro JL, Costa CA, Janssen LP. Evaluating the use of chatbot during pregnancy: A usability study. Healthcare Analytics. 2022;2(100072):1-9.
Campos-Filho AS, Cursino JR, Barros-Júnior TD, Lima EC. Assistente Virtual na Educação em Saúde dos Homens. J Health Inform. 2023;15(Esp):1-14.
Luykx JJ, Gerritse F, Habets PC, Vinkers CH. The performance of ChatGPT in generating answers to clinical questions in psychiatry: a two-layer assessment. World Psychiatry. 2023;22(3):479-480.
Abdullahi T, Singh R, Eickhoff C. Learning to make rare and complex diagnoses with generative ai assistance: qualitative study of popular large language models. JMIR Medical Education. 2024;10(1), 1-11.
Spallek S, Birrell L, Kershaw S, Devine EK, Thornton L. Can we use chatgpt for mental health and substance use education? examining its quality and potential harms. JMIR Medical Education. 2023;9(1), 1-10.
Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv. 2023:2(2303.13375), 1-35.
Lautrup AD, Hyrup T, Schneider-Kamp A, Dahl M, Lindholt JS, Schneider-Kamp P. Heart-to-heart with ChatGPT: the impact of patients consulting AI for cardiovascular health advice. Open Heart. 2023;10(2), 1-8.
Andrew A. Potential applications and implications of large language models in primary care. Fam Med Community Health, 2024;12(Suppl 1), 1-6.
Corrêa JS, Araújo-Neto AP, Pinto GR, Lima LD, Teles AS. Lhia: a smart chatbot for breastfeeding education and recruitment of human milk donors. Appl Sci. 2023;13(12): 1-19
Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, et al. A survey of large language models. arXiv preprint arXiv: 2023:13(2303.18223), 1-124.
Espejel JL, Ettifouri EH, Alassan MS, Chouham EM, Dahhane W. GPT-3.5, GPT-4, or BARD? evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts. Natural Language Processing Journal. 2023;5(100032), 1-192.
Grandini M, Bagli E, Visani G. Metrics for multi-class classification: an overview. arXiv preprint arXiv: 2020:1(2008.05756), 1-17.
Jedrzejczak WW, Kochanek K. Comparison of the audiological knowledge of three chatbots-ChatGPT, Bing Chat, and Bard. Audiol Neurootol. 2023:11(38710158), 1-7.
Lim ZW, Pushpanathan K, Yew SM, Lai Y, Sun CH, Lam JS, et al. Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine. 2023:95(104770), 1-11.
Kim SH, Schramm S, Berberich C, Rosenkranz E, Schmitzer L, Serguen K, et al. Human-AI collaboration in large language model-assisted brain mri differential diagnosis: a usability study. medRxiv 2024:02(05), 1-22.
Simas WL, Penha JS, Soares LB, Rabêlo PP, Oliveira BL, Pinheiro FS. Insegurança materna na amamentação em lactantes atendidas em um banco de leite humano. Rev. Bras. Saude Mater. Infant. 2021:21(1), 251-259.
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Submission of a paper to Journal of Health Informatics is understood to imply that it is not being considered for publication elsewhere and that the author(s) permission to publish his/her (their) article(s) in this Journal implies the exclusive authorization of the publishers to deal with all issues concerning the copyright therein. Upon the submission of an article, authors will be asked to sign a Copyright Notice. Acceptance of the agreement will ensure the widest possible dissemination of information. An e-mail will be sent to the corresponding author confirming receipt of the manuscript and acceptance of the agreement.
