Chatbots in identification of breastfeeding issues: performance evaluation
DOI:
https://doi.org/10.59681/2175-4411.v16.iEspecial.2024.1370Keywords:
Breastfeeding, Artificial Intelligence, Expert SystemsAbstract
Objective: This study aimed to evaluate the performance of artificial intelligence-based chatbots in identifying breastfeeding-related problems. Method: The study assessed OpenAI ChatGPT-3.5, Microsoft Copilot, Google Gemini, and Lhia in identifying breastfeeding issues. Lhia chatbot is being developed by our team of researchers. Through consensus among healthcare professionals specializing in breastfeeding, a dataset of annotated main clinical complaint reports from medical records at the University Hospital of the Federal University of Maranhão was created for testing with three zero-shot prompt approaches. Results: The best performance was achieved by ChatGPT-3.5, which demonstrated accuracy ranging from 79% to 93%, fallback from 0% to 7%, and F1-score from 75% to 100%. Conclusion: Artificial intelligence-based chatbots can be a promising tool to assist mothers and healthcare professionals in the early detection of breastfeeding issues.
References
Victora CG, Bahl R, Barros AJ, França GV, Horton S, Krasevec J, et al. Breastfeeding in the 21st century: epidemiology, mechanisms, and lifelong effect. The Lancet. 2016;387(10017):475-90. DOI: https://doi.org/10.1016/S0140-6736(15)01024-7
Rollins NC, Bhandari N, Hajeebhoy N, Horton S, Lutter CK, Martines JC, et al. Breastfeeding in the 21st century: epidemiology, mechanisms, and lifelong effect. Lancet. 2016;387(10017):491-504 DOI: https://doi.org/10.1016/S0140-6736(15)01044-2
World Health Organization. Infant and Young Child Feeding: Model Chapter for Textbooks for Medical Students and Allied Health Professionals; Technical Report; WHO: Geneva, Switzerland, 2009
Bhattacharjee NV, Schaeffer LE, Hay SI, Lu D, Schipp MF, Lazzar-Atwood A, et al. Mapping inequalities in exclusive breastfeeding in low- and middle-income countries 2000–2018. Nat Hum Behav. 2021; 5,1027–1045.
Softić A, Husić JB, Softić A, Baraković S. Health chatbot: design, implementation, acceptance and usage motivation. In 20th International Symposium Infoteh-Jahorina; 2021 17-19 March; East Sarajevo, Bosnia and Herzegovina. IEEE; 2021, pp. 1-6, Available from: doi:10.1109/INFOTEH51037.2021.9400693. DOI: https://doi.org/10.1109/INFOTEH51037.2021.9400693
Prasad VA, Ranjith R. Intelligent chatbot for lab security and automation. In 11th international conference on computing, communication and networking Technologies, 2020 1-3 July; Kharagpur, India. IEEE, 2020, pp. 1-4, Available from: doi:10.1109/ICCCNT49239.2020.9225641.
Yadav D, Malik P, Dabas K, Singh P. Feedpal: Understanding Opportunities for Chatbots in Breastfeeding Education of Women in India. Proceedings of the ACM on Human-Computer Interaction. 2019;3(CSCW):170:1–30. DOI: https://doi.org/10.1145/3359272
Gupta V, Arora N, Jain Y, Mokashi S, Panda C. Assessment on Adoption Behavior of First-time Mothers on the Usage of Chatbots for Breastfeeding Consultation. J Mahatma Gandhi Univ Med Sci Tech. 2021;6(2):64-68. DOI: https://doi.org/10.5005/jp-journals-10057-0161
Montenegro JL, Costa CA, Janssen LP. Evaluating the use of chatbot during pregnancy: A usability study. Healthcare Analytics. 2022;2(100072):1-9. DOI: https://doi.org/10.1016/j.health.2022.100072
Campos-Filho AS, Cursino JR, Barros-Júnior TD, Lima EC. Assistente Virtual na Educação em Saúde dos Homens. J Health Inform. 2023;15(Esp):1-14. DOI: https://doi.org/10.59681/2175-4411.v15.iEspecial.2023.1087
Luykx JJ, Gerritse F, Habets PC, Vinkers CH. The performance of ChatGPT in generating answers to clinical questions in psychiatry: a two-layer assessment. World Psychiatry. 2023;22(3):479-480. DOI: https://doi.org/10.1002/wps.21145
Abdullahi T, Singh R, Eickhoff C. Learning to make rare and complex diagnoses with generative ai assistance: qualitative study of popular large language models. JMIR Medical Education. 2024;10(1), 1-11. DOI: https://doi.org/10.2196/51391
Spallek S, Birrell L, Kershaw S, Devine EK, Thornton L. Can we use chatgpt for mental health and substance use education? examining its quality and potential harms. JMIR Medical Education. 2023;9(1), 1-10. DOI: https://doi.org/10.2196/51243
Nori H, King N, McKinney SM, Carignan D, Horvitz E. Capabilities of gpt-4 on medical challenge problems. arXiv preprint arXiv. 2023:2(2303.13375), 1-35.
Lautrup AD, Hyrup T, Schneider-Kamp A, Dahl M, Lindholt JS, Schneider-Kamp P. Heart-to-heart with ChatGPT: the impact of patients consulting AI for cardiovascular health advice. Open Heart. 2023;10(2), 1-8. DOI: https://doi.org/10.1136/openhrt-2023-002455
Andrew A. Potential applications and implications of large language models in primary care. Fam Med Community Health, 2024;12(Suppl 1), 1-6. DOI: https://doi.org/10.1136/fmch-2023-002602
Corrêa JS, Araújo-Neto AP, Pinto GR, Lima LD, Teles AS. Lhia: a smart chatbot for breastfeeding education and recruitment of human milk donors. Appl Sci. 2023;13(12): 1-19 DOI: https://doi.org/10.3390/app13126923
Zhao WX, Zhou K, Li J, Tang T, Wang X, Hou Y, et al. A survey of large language models. arXiv preprint arXiv: 2023:13(2303.18223), 1-124.
Espejel JL, Ettifouri EH, Alassan MS, Chouham EM, Dahhane W. GPT-3.5, GPT-4, or BARD? evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts. Natural Language Processing Journal. 2023;5(100032), 1-192. DOI: https://doi.org/10.1016/j.nlp.2023.100032
Grandini M, Bagli E, Visani G. Metrics for multi-class classification: an overview. arXiv preprint arXiv: 2020:1(2008.05756), 1-17.
Jedrzejczak WW, Kochanek K. Comparison of the audiological knowledge of three chatbots-ChatGPT, Bing Chat, and Bard. Audiol Neurootol. 2023:11(38710158), 1-7. DOI: https://doi.org/10.1159/000538983
Lim ZW, Pushpanathan K, Yew SM, Lai Y, Sun CH, Lam JS, et al. Benchmarking large language models’ performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard. EBioMedicine. 2023:95(104770), 1-11. DOI: https://doi.org/10.1016/j.ebiom.2023.104770
Kim SH, Schramm S, Berberich C, Rosenkranz E, Schmitzer L, Serguen K, et al. Human-AI collaboration in large language model-assisted brain mri differential diagnosis: a usability study. medRxiv 2024:02(05), 1-22. DOI: https://doi.org/10.1101/2024.02.05.24302099
Simas WL, Penha JS, Soares LB, Rabêlo PP, Oliveira BL, Pinheiro FS. Insegurança materna na amamentação em lactantes atendidas em um banco de leite humano. Rev. Bras. Saude Mater. Infant. 2021:21(1), 251-259. DOI: https://doi.org/10.1590/1806-93042021000100013
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Submission of a paper to Journal of Health Informatics is understood to imply that it is not being considered for publication elsewhere and that the author(s) permission to publish his/her (their) article(s) in this Journal implies the exclusive authorization of the publishers to deal with all issues concerning the copyright therein. Upon the submission of an article, authors will be asked to sign a Copyright Notice. Acceptance of the agreement will ensure the widest possible dissemination of information. An e-mail will be sent to the corresponding author confirming receipt of the manuscript and acceptance of the agreement.
