An evaluation framework for information systems from the users’ perspective: a scoping review
DOI:
https://doi.org/10.59681/2175-4411.v18.2026.1539Keywords:
Evaluation models, Information System, Information TechnologyAbstract
This scoping review aims to synthesize scientific literature on information systems evaluation methods from the users’ perspective, focusing on their characteristics, construction stages, and measurement properties. The search was conducted in 2024 across the following databases: ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, LILACS, MEDLINE (via PubMed), Scopus, and Web of Science. The study selection process was independently carried out by two reviewers. A total of 18,837 articles were identified in the searches, and 11 were ultimately included in the review. The studies presented various evaluation frameworks specifically for health information systems, with one study also evaluating information systems from other fields. Only five studies reported the frameworks’ validation procedures, and very few provided comprehensive results of the validations or sufficiently thorough analyses. This scoping review underscores the methodological diversity in information systems evaluation models and highlights the limitations of current use. It emphasizes the need for standardized approaches to enhance comparability, transparency, and long-term system development.
Downloads
References
Pleskach V, Pleskach M, Zelikovska O. Information security management system in distributed information systems. In: 2019 IEEE International Conference on Advanced Trends in Information Theory (ATIT); 2019. p. 300-3. doi: https://doi.org/10.1109/ATIT49449.2019.9030484
Gregersen E. 5 components of information systems. Encyclopaedia Britannica; 2021.
Stoykova S, Shakev N. Artificial intelligence for management information systems: opportunities, challenges, and future directions. Algorithms. 2023;16(8):357. doi: https://doi.org/10.3390/a16080357
Maia MAQ, Barbosa RR, Williams P. Usabilidade e experiência do usuário de sistemas de informação: em busca de limites e relações. Ciênc Inf Rev. 2020;6(3):34-48. doi: https://doi.org/10.28998/cirev.2019v6n3c
BSI. ISO 9241-210:2019. Ergonomics of human-system interaction. Part 210: Human-centred design for interactive systems. London: BSI; 2019.
Cintho LM, Machado RR, Moro CMC. Métodos para avaliação de sistema de informação em saúde. J Health Inform. 2016;8(2). Disponível em: https://jhi.sbis.org.br/index.php/jhi-sbis/article/view/346
Osborne SE. Evaluating health care information systems: methods and applications. J Adv Nurs. 1994;20(1):203-4. doi: https://doi.org/10.1046/j.1365-2648.1994.20010202-6.x
Morais RM, Costa AL. Um modelo para avaliação de sistemas de informação do SUS de abrangência nacional: o processo de seleção e estruturação de indicadores. Rev Adm Pública. 2014;48(3):767-93. doi: https://doi.org/10.1590/0034-76121512
Garcia-Smith D, Effken JA. Development and initial evaluation of the Clinical Information Systems Success Model (CISSM). Int J Med Inform. 2013;82(6):539-52. doi: https://doi.org/10.1016/j.ijmedinf.2013.01.011
Gürsel G, Zayim N, Gülkesen KH, Arifoğlu A, Saka O. A new approach in the evaluation of hospital information systems. Turk J Elec Eng Comp Sci. 2014;22(1):214-22. doi: https://doi.org/10.3906/elk-1110-15
Kanungo S, Duda S, Srinivas Y. A structured model for evaluating information systems effectiveness. Syst Res Behav Sci. 1999;16(6):495-518. doi: https://doi.org/10.1002/(SICI)1099-1743(199911/12)16:6<495::AID-SRES238>3.0.CO;2-R
Kukhareva PV, Weir C, Del Fiol G, Aarons GA, Taft TY, Schlechter CR, et al. Evaluation in Life Cycle of Information Technology (ELICIT) framework: supporting the innovation life cycle from business case assessment to summative evaluation. J Biomed Inform. 2022;127:104014. doi: https://doi.org/10.1016/j.jbi.2022.104014
Lau F, Hagens S, Muttitt S. A proposed benefits evaluation framework for health information systems in Canada. Healthc Q. 2007;10(1):112-6, 118.
Norouzi H, Mehrolhassani MH, Hajesmaeel-Gohari S, Ahmadian L, Ghaemi MM, Mohammadi M, et al. Identification and classification of indicators for evaluating health information systems. Health Policy Technol. 2024;13(2):100833. doi: https://doi.org/10.1016/j.hlpt.2023.100833
Sockolow PS, Weiner JP, Bowles KH, Lehmann HP. A new instrument for measuring clinician satisfaction with electronic health records. Comput Inform Nurs. 2011;29(10):574-85. doi: https://doi.org/10.1097/NCN.0b013e31821a1568
Stockdale R, Standing C. An interpretive approach to evaluating information systems: a content, context, process framework. Eur J Oper Res. 2006;173(3):1090-102. doi: https://doi.org/10.1016/j.ejor.2005.07.006
Stylianides A, Mantas J, Roupa Z, Yamasaki EN. Development of an evaluation framework for health information systems (DIPSA). Acta Inform Med. 2018;26(4):230-4. doi: https://doi.org/10.5455/aim.2018.26.230-234
Yusof MM, Kuljis J, Papazafeiropoulou A, Stergioulas LK. An evaluation framework for health information systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform. 2008;77(6):386-98. doi: https://doi.org/10.1016/j.ijmedinf.2007.08.011
Reichenheim M, Bastos JL. What, what for and how? Developing measurement instruments in epidemiology. Rev Saude Publica. 2021;55:40. doi: https://doi.org/10.11606/s1518-8787.2021055002813
Barzekar H, Ebrahimzadeh F, Luo J, Karami M, Robati Z, Goodarzi P. Adoption of hospital information system among nurses: a technology acceptance model approach. Acta Inform Med. 2019;27(5):305-10. doi: https://doi.org/10.5455/aim.2019.27.305-310
Ammenwerth E. Technology acceptance models in health informatics: TAM and UTAUT. Stud Health Technol Inform. 2019;263:64-71. doi: https://doi.org/10.3233/SHTI190111
Rana RH, Alam K, Gow J. Health outcome and expenditure in low-income countries: does increasing diffusion of information and communication technology matter? Inf Technol Dev. 2020;26(3):506-24. doi: https://doi.org/10.1080/02681102.2019.1678455
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Vanessa Pereira Corrêa Rampinelli, Catiele Raquel Schmidt, Júlia Meller Dias de Oliveira, Ianka Cristina Celuppi, Arthur Thives Mello, Jades Fernando Hammes, Célio Luiz Cunha, Raul Sidnei Wazlawick, Danielle Soares Rocha Vieira, Eduardo Monguilhott Dalmarco

This work is licensed under a Creative Commons Attribution 4.0 International License.