Optimizing internal auditing of surgical records: an AI approach

Authors

DOI:

https://doi.org/10.59681/2175-4411.v17.2025.1478

Keywords:

Hospital audit, Artifical Intelligence, Machine Learning

Abstract

Objective: The aim was to develop a Generative Artificial Intelligence-based hospital audit assistant capable of analyzing and classifying information from surgical medical records, in order to support internal auditing procedures at the Hospital das Clínicas of the Federal University of Pernambuco (HC-UFPE). Methods: The methodology involved training a large language model (LLM) with expert-validated data, followed by performance evaluation using sensitivity, specificity, Kappa index, and Area Under the ROC Curve (AUC-ROC). The database utilized was the Management Application for University Hospitals (AGHU), from which data was extracted in JSON format after pre-processing and treatment using Metabase software. The OpenAI ecosystem was adopted through the customization of ChatGPT, GPT-4o model. We applied prompt engineering followed by fine-tuning techniques during the training process. Results: The virtual assistant developed for hospital auditing demonstrated a sensitivity of 96.8%, specificity of approximately 77%, a Kappa index of 0.78, and AUC-ROC of 86.95%, indicating that the model effectively analyzes medical records with high precision. The results showed that the assistant could effectively replicate human evaluations, demonstrating significant potential for transforming hospital audit practices. Conclusion: It is concluded that the hospital audit assistant can optimize internal processes for the proposed sector, enhancing support, safety, accuracy, and comprehensiveness in medical record auditing.

Downloads

Download data is not yet available.

Author Biographies

Rita de Cássia Almeida Sales, Universidade Federal de Pernambuco

Professional with solid academic background and extensive experience in the health area, with emphasis on systems auditing and hospital management. Master's student in Biomedical Engineering from the Federal University of Pernambuco (UFPE), with specializations in Health Systems Auditing, Nephrology Nursing, Intensive Care Nursing, Conflict Mediation and Nonviolent Communication, in addition to an MBA in Hospital Management. My career is marked by working as a clinical nurse and coordinating teams in specialized health units (Nephrology). Over the years, I have developed in-depth knowledge of care practices, with emphasis on the areas of Nephrology, Hospital Auditing and Health Systems Management, always focused on improving the quality of patient care. I currently work in the Information Technology and Digital Health (SETISD) and Contracting and Regulation (STCOR) sector of HC-UFPE, where I contribute directly to the implementation of technological innovations in the hospital environment, including the development of data-based control panels and management reports. My complementary training includes several short courses in areas such as hospital planning and management, integrative and complementary practices, digital health, among others. This diversity of qualifications allows me to have a broad and strategic vision of health, combining technical knowledge with the ability to lead and implement innovative projects.

Isaura Romero Peixoto, Universidade Federal de Pernambuco

Graduated in Medicine from the Federal University of Pernambuco (UFPE). Master in Medicine from UFPE and PhD in Medicine from UFPE. She is currently a preceptor of the Medical Residency in Geriatrics at the Hospital das Clínicas (HC) of UFPE, Geriatrician at the Elderly Care Center (NAI/UFPE), Auditor Doctor at HC-UFPE and visiting professor in the Postgraduate Program in Gerontology. She has experience in the area of ​​Medicine, with emphasis on Geriatrics/Gerontology and Palliative Care. Professor of the undergraduate program in Medicine at AFYA (Faculty of Medical Sciences of Jaboatão). She is motivated and interested in working in Geriatrics and Higher Education using active and traditional methodologies, with qualification in health simulation and PBL.

Shirley da Silva Jacinto de Oliveira Cruz, Empresa Brasileira de Serviços Hospitalares

Graduated in Computer Science from the Federal University of Pernambuco (2008), master's degree (2010) and doctorate (2016) from the same institution, in addition to having been a visiting researcher at the University of Western Ontario (UWO) (2012). Currently, she is an Information Technology Analyst at UFPE, acting as head of the IT and Digital Health sector at Hospital das Clínicas (HC-UFPE/ EBSERH). She has experience in the area of ​​innovation and digital transformation. She worked as Systems Development Manager at the Information Technology Center (NTI-UFPE) from 2008 to 2013, as ICT Governance Coordinator at NTI-UFPE from 2013 to 2015, as ICT Governance and Process Management Director from 2015 to 2019 at the Pro-Rectory of Communication, Information and Information Technology (PROCIT-UFPE) and as Coordinator of Articulation and Partnerships at the Pro-Rectory of Research and Innovation (PROPESQI-UFPE).

Wellington Pinheiro dos Santos, Universidade Federal de Pernambuco

Wellington Pinheiro dos Santos holds a degree in Electrical and Electronic Engineering (2001) and a master's degree in Electrical Engineering (2003) from the Federal University of Pernambuco, and a PhD in Electrical Engineering from the Federal University of Campina Grande (2009). He is currently an Associate Professor (full-time) in the Department of Biomedical Engineering of the Center for Technology and Geosciences - School of Engineering of Pernambuco, Federal University of Pernambuco, working in the Undergraduate Program in Biomedical Engineering and in the Graduate Program in Biomedical Engineering, of which he was one of the founders (2011). He founded the Center for Social Technologies and Bioengineering of the Federal University of Pernambuco, NETBio-UFPE (2012). He is also a member of the Postgraduate Program in Computer Engineering at the Polytechnic School of Pernambuco, University of Pernambuco, since 2009. He is the coordinator of the Biomedical Computing and Bioengineering Axis of the National Institute of Science and Technology of the Economic-Industrial Complex of Health - iCEIS. He has experience in the areas of Biomedical Engineering and Computer Science, with an emphasis on Graphics Processing, working mainly on the following topics: digital image processing, pattern recognition, computer vision, evolutionary computation, numerical optimization methods, computational intelligence, image formation techniques, virtual reality, game design and applications of Computing and Engineering in Medicine and Biology. He is a member of the Brazilian Society of Biomedical Engineering (SBEB), the Brazilian Society of Computational Intelligence (SBIC, formerly SBRN), and the International Federation of Medical and Biological Engineering (IFMBE).

References

Pinheiro MB, Campos RKGG, Maniva SJC de F, Rolim KMC, Bonfim IM. Tecnologias disponíveis para o processo de auditoria interna em classificação de risco: revisão integrativa. Rev Bras Pesqui Em SaúdeBrazilian J Health Res. 2023;25(4):81–8.

Jacomini LDS, Mangiavacchi BM. AUDITORIA HOSPITALAR COMO INSTRUMENTO NA PREVENÇÃO DE ERROS MEDICAMENTOSOS. Múltiplos Acessos. 9 de junho de 2021;6(1):199–207.

Ceretta JC, Seibert RM, Callegaro ARC. Gestão hospitalar: a auditoria operacional como ferramenta estratégica para o controle de desperdícios. Rev Gest E Secr. 1o de março de 2023;14(3):2663–75.

Pimentel LCL, Virginio JPA, Albernaz CB, Souza DP de, Jesus CAC de, Paranaguá TT de B, et al. Avaliação da qualidade dos registros do processo de enfermagem por meio de auditoria retrospectiva. Rev Enferm UERJ. 12 de dezembro de 2023;31:e77316–e77316.

Caixeta FDC, Silva AA, Ferreira MV, Sales RDCA. AUDITORIA EM SAÚDE: NÃO CONFORMIDADES EM REGISTROS DE PRONTUÁRIOS CIRÚRGICOS. Int J Health Sci. 30 de outubro de 2024;3(2):181–2.

White J, Fu Q, Hays S, Sandborn M, Olea C, Gilbert H, et al. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT [Internet]. arXiv; 2023 [citado 25 de março de 2025]. Disponível em: http://arxiv.org/abs/2302.11382

Liu NF, Lin K, Hewitt J, Paranjape A, Bevilacqua M, Petroni F, et al. Lost in the Middle: How Language Models Use Long Contexts [Internet]. arXiv; 2023 [citado 25 de março de 2025]. Disponível em: http://arxiv.org/abs/2307.03172

An S, Ma Z, Lin Z, Zheng N, Lou JG. Make Your LLM Fully Utilize the Context [Internet]. arXiv; 2024 [citado 25 de março de 2025]. Disponível em: http://arxiv.org/abs/2404.16811

Baker GA, Raut A, Shaier S, Hunter LE, Wense K von der. Lost in the Middle, and In-Between: Enhancing Language Models’ Ability to Reason Over Long Contexts in Multi-Hop QA [Internet]. arXiv; 2024 [citado 25 de março de 2025]. Disponível em: http://arxiv.org/abs/2412.10079

Chang Y, Wang X, Wang J, Wu Y, Yang L, Zhu K, et al. A Survey on Evaluation of Large Language Models [Internet]. arXiv; 2023 [citado 25 de março de 2025]. Disponível em: http://arxiv.org/abs/2307.03109

Verspoor K. “Fighting fire with fire” - using LLMs to combat LLM hallucinations. Nature. junho de 2024;630(8017):569–70.

Downloads

Published

2025-12-26

How to Cite

de Cássia Almeida Sales, R., Romero Peixoto, I., da Silva Jacinto de Oliveira Cruz, S., & Pinheiro dos Santos, W. (2025). Optimizing internal auditing of surgical records: an AI approach. Journal of Health Informatics, 17(1), 1478. https://doi.org/10.59681/2175-4411.v17.2025.1478

Issue

Vol. 17 (2025)

Section

Original Articles

License

Copyright (c) 2025 Rita de Cássia Almeida Sales, Isaura Romero Peixoto, Shirley da Silva Jacinto de Oliveira Cruz, Wellington Pinheiro dos Santos

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Similar Articles

1 2 3 4 5 6 7 8 9 10 > >> 

You may also start an advanced similarity search for this article.