Rampinelli VPC et al.
J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
J Health Inform. 2026;18:e1539
DOI: https://doi.org/10.59681/2175-4411.v18.2026.1539
Review Article

An evaluation framework for information systems from the users’ perspective: a scoping review

Estrutura de avaliação para sistemas de informação na perspectiva dos usuários: uma revisão de escopo
Un marco de evaluación para sistemas de información desde la perspectiva de los usuarios: una revisión de alcance
Vanessa Pereira Corrêa Rampinelli1iD, Catiele Raquel Schmidt1iD, Júlia Meller Dias de Oliveira1iD, Ianka Cristina Celuppi1iD, Arthur Thives Mello1iD, Jades Fernando Hammes2iD, Célio Luiz Cunha2iD, Raul Sidnei Wazlawick1iD, Danielle Soares Rocha Vieira1iD, Eduardo Monguilhott Dalmarco1iD
2 Laboratório Bridge, Florianópolis (SC), Brazil.

Article information

Type Review Article
Identifier e1539
DOI 10.59681/2175-4411.v18.2026.1539
Received 8 September 2025
Accepted 16 April 2026
Published20 May 2026
Review process Double Blind Review
Handling editorJuliano de Souza Gaspar
Access Open, CC BY 4.0
Corresponding authorVanessa Pereira Corrêa Rampinelli
fisio.vpc@gmail.com

License

This is an open access article published under the Creative Commons CC BY 4.0 license, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited.
CC BY 4.0
Abstract
This scoping review aimed to synthesize scientific literature on information systems evaluation methods from the users’ perspective, focusing on their characteristics, construction stages, and measurement properties. The search was conducted in 2024 across the following databases: ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, LILACS, MEDLINE (via PubMed), Scopus, and Web of Science. The study selection process was carried out independently by two reviewers. A total of 18,837 articles were identified, and 11 were ultimately included in the review. The studies presented various evaluation frameworks specific to health information systems, with one study also evaluating information systems from other fields. Only five studies reported the frameworks' validation procedures, and very few provided comprehensive validation results or sufficiently thorough analyses. This scoping review underscores the methodological diversity in information systems evaluation models and highlights the limitations of current use. It emphasizes the need for standardized approaches to enhance comparability, transparency, and long-term system development.
Keywords: Evaluation models; Information systems; Information technology.
Resumo
Esta revisão de escopo teve como objetivo sintetizar a literatura científica sobre estruturas de avaliação de sistemas de informação sob a perspectiva dos usuários, com foco em suas características, etapas de construção e propriedades de mensuração. A busca foi realizada em 2024 nas seguintes bases de dados: ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, LILACS, MEDLINE (via PubMed), Scopus e Web of Science. O processo de seleção dos estudos foi realizado de forma independente por dois revisores. Um total de 18.837 artigos foram identificados nas buscas, e 11 foram incluídos na revisão. Os estudos apresentaram diversas estruturas de avaliação específicas para sistemas de informação em saúde, com um estudo também avaliando sistemas de informação de outras áreas. Apenas cinco estudos relataram os procedimentos de validação das estruturas, e poucos forneceram resultados abrangentes das validações ou análises suficientemente completas. Esta revisão de escopo destaca a diversidade metodológica nos modelos de avaliação de sistemas de informação e destaca as limitações do uso atual. Enfatiza a necessidade de abordagens padronizadas para aprimorar a comparabilidade, a transparência e o desenvolvimento de sistemas a longo prazo.
Descritores: Modelos de avaliação; Sistemas de informação; Tecnologia da informação.
Resumen
Esta revisión de alcance busca sintetizar la literatura científica sobre métodos de evaluación de sistemas de información desde la perspectiva de los usuarios, centrándose en sus características, etapas de construcción y propiedades de medición. La búsqueda se realizó en 2024 en las siguientes bases de datos: ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, LILACS, MEDLINE (vía PubMed), Scopus y Web of Science. El proceso de selección de estudios fue realizado de forma independiente por dos revisores. Se identificaron 18.837 artículos en las búsquedas, de los cuales 11 se incluyeron en la revisión. Los estudios presentaron diversos marcos de evaluación específicos para sistemas de información en salud, y un estudio también evaluó sistemas de información de otros campos. Solo cinco estudios informaron sobre los procedimientos de validación de los marcos, y muy pocos proporcionaron resultados completos de las validaciones o análisis suficientemente exhaustivos. Esta revisión de alcance subraya la diversidad metodológica de los modelos de evaluación de sistemas de información y destaca las limitaciones de su uso actual. Enfatiza la necesidad de enfoques estandarizados para mejorar la comparabilidad, la transparencia y el desarrollo de sistemas a largo plazo.
Descriptores: Modelos de evaluación; Sistemas de información; Tecnología de la información.
HOW TO CITE THIS ARTICLE: Rampinelli VPC, Schmidt CR, Oliveira JMD, Celuppi IC, Mello AT, Hammes JF, et al. An evaluation framework for information systems from the users’ perspective: a scoping review. J Health Inform 2026;18:e1539. doi: https://doi.org/10.59681/2175-4411.v18.2026.1539
https://doi.org/10.59681/2175-4411.v18.2026.15391
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539

1 Introduction

Information systems (IS) are organized sets of components designed to collect, process, store, and distribute information. They support decision-making, coordination, control, analysis, and visualization within organizations [1]. These components include hardware, software, networks, data, and personnel, which work together to streamline operations and deliver relevant information [2]. Health information systems (HIS) are well established within this domain, and IS also play key roles in public sectors such as education, mobility, security, and geolocation.

Evaluating the quality of IS is complex due to the multitude of factors involved. Deploying these systems across various sectors introduces further challenges in aligning multidimensional quality aspects with specific user needs. Understanding both their advantages and challenges, especially from the user’s perspective, is essential [3]. IS use refers to how individuals apply computer systems in their professional activities, reflecting both adoption and effectiveness.

Users, whether individuals, groups, or organizations, interact with these systems to perform tasks, access information, conduct transactions, or make decisions. They can be categorized as end, administrative, technical, managerial, or external users [4]. End users rely on the system for daily tasks; examples include healthcare professionals using electronic health records (EHR) or employees accessing enterprise management systems. Defining user categories is essential for designing, implementing, and evaluating systems, as each group’s needs and expectations influence usability, efficiency, and overall success [5].

Despite the importance of user-centered evaluation, few studies present comprehensive frameworks from this perspective. Most available studies come from the health sector or are over thirty years old [6,7]. Many focus on technical or organizational aspects rather than the user experience, prioritizing performance, implementation, or economic impacts over acceptance, satisfaction, and ease of use. Evaluations also tend to be system-specific, limiting generalizability. Thus, a significant gap remains regarding frameworks that incorporate the user perspective.

This manuscript examines key sectors of national governance: the economy, agriculture, health, education, the environment, and infrastructure, which are essential for resource management, policy formulation, and addressing societal needs equitably and sustainably. IS in these areas support data collection, analysis, and dissemination, thereby improving decision-making, transparency, and efficiency. This study addresses the current gap in the literature by synthesizing IS evaluation methods from the user perspective, with a specific focus on the public sector. Accordingly, this scoping review aimed to

synthesize the scientific literature on evaluating IS from a user perspective, examining the characteristics, development stages, and measurement properties of these evaluation frameworks.

2 Methods

This scoping review was conducted in accordance with Joanna Briggs Institute guidelines and the PRISMA Extension for Scoping Reviews (PRISMA-ScR). The study protocol was registered on the Open Science Framework (https://doi.org/10.17605/OSF.IO/H89T2). In August 2024, a comprehensive search was conducted across several databases, specifically: the ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, Scientific Health Information from Latin America and the Caribbean countries (LILACS), Medical Literature Analysis and Retrieval System Online (MEDLINE) via PubMed, Scopus, and Web of Science (Table 1). The search strategy focused on three main concepts: “information systems,” “information technology,” and “evaluation framework.” Initially, the strategy was implemented in MEDLINE (via PubMed); the search terms were subsequently adapted to meet the specific requirements of each database. Additionally, two separate searches were conducted on Google Scholar, one in English and the other in Portuguese, screening only the first 100 articles from each search. No restrictions or filters were applied during the database searches. The retrieved articles were then exported to Rayyan® (Qatar Computing Research Institute, Qatar).

Afterward, Reviewer 1 excluded duplicate records by utilizing the platform’s automatic deduplication feature, followed by manual verification. Subsequently, Reviewer 2 was added to the Rayyan project with the blinding feature enabled. In the first eligibility phase, titles and abstracts were screened using the inclusion criteria (Table 2). Initially, the two reviewers independently excluded 10 articles, followed by a consensus meeting. Thereafter, all titles and abstracts were screened independently, and disagreements were resolved by consensus. In the second eligibility phase, full texts were evaluated using a similar approach of independent assessment and consensus. Full texts that were not readily available were requested from the corresponding authors via three attempts over a three-week period. The reference lists of the included full texts were also reviewed.

There were no restrictions regarding the language, geographic location, or publication date of the studies. However, gray literature (e.g., letters, book chapters, non-peer-reviewed conference papers, opinion pieces, articles describing techniques, panels, preprints, and clinical or technical guides) was excluded. Full texts that remained unavailable after three attempts to contact the corresponding authors over a three-week

https://doi.org/10.59681/2175-4411.v18.2026.15392
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
Table 1. Search strategy in the databases (search on August 31, 2024).
DatabaseSearch strategyRecords
ACM Digital Library[[Abstract: "information systems"] OR [Abstract: "information system"] OR [Abstract: technology] OR [Abstract: technologies]] AND [[Abstract: "evaluation framework"] OR [Abstract: "evaluation frameworks"] OR [Abstract: "structured evaluation"] OR [Abstract: "framework development"] OR [Abstract: "software validation"] OR [Abstract: " software validity "] OR [Abstract: "software verification"]]693
Embase(('information system'/de OR 'information systems':ti,ab,kw OR 'information system*':ti,ab,kw OR 'information technology'/de OR 'technology'/de OR technolog*:ti,ab,kw) AND 'evaluation framework':ti,ab,kw OR 'evaluation framework*':ti,ab,kw OR 'structured evaluation':ti,ab,kw OR 'framework development':ti,ab,kw OR 'software validat*':ti,ab,kw OR 'software validation':ti,ab,kw OR 'software verificat*':ti,ab,kw) AND [embase]/lim2732
Engineering Village((({Information Systems} WN CV) OR ({information management} WN CV) OR ({management information systems} WN CV) OR ({Information System*} WN KY) OR ({Information Technolog*} WN KY) OR ({35} WN ICS) OR ({Technolog*} WN KY))) AND ( (({Evaluation framework*} WN KY) OR ({structured evaluat*} WN KY) OR ({Framework Development} WN KY) OR ({software validat*} WN KY) OR ({software verificat*} WN KY) OR ({Software quality} WN CV) OR ({Computer software selection and evaluation} WN CV) OR ({03.120.01} WN ICS) OR ({03.120.99} WN ICS)))4888
ERIC(descriptorx:"Information Systems" OR descriptorx:"Information Technology" OR descriptorx:"Technology" OR descriptorx:"Educational Technology" OR title:"information system" OR title:"technology" OR abstract:"information system" OR abstract:"technology") AND (descriptorx:"Computer Software Evaluation" OR descriptorx:"Computer Software Reviews" OR title:"Evaluation framework" OR title:"structured evaluation" OR title:"Framework Development" OR title:"software validation" OR title:"software verification" OR abstract:"Evaluation framework" OR abstract:"structured evaluation" OR abstract:"Framework Development" OR abstract:"software validation" OR abstract:"software verification")1436
IEEE Xplore("All Metadata":"Information System*" OR "All Metadata":"Information Technolog*" OR "All Metadata":Technolog*) AND ("All Metadata":"Evaluation framework" OR "All Metadata":"structured evaluat*" OR "All Metadata":"Framework Development" OR "All Metadata":"software validat*" OR "All Metadata":"software verificat*")2157
LILACS(tw:("Information Systems" OR "information system" OR "information systems" OR Technology OR technologies OR "Sistemas de Informação" OR "Sistema de Informação" OR "ICT" OR "TIC" OR "Tecnologia" OR "Tecnologias" OR "Sistema Tecnológico" OR "Sistemas Tecnológicos" OR "Sistemas de Información" OR "sistema de información" OR Tecnología OR Tecnologías OR "Systèmes de recherche d'information" OR "Systèmes d'information" OR Technologie OR Technologies)) AND (tw:("Evaluation framework" OR "Evaluation frameworks" OR "structured evaluation" OR "Framework Development" OR "Modelo de avaliação" OR "Estrutura de avaliação" OR "Avaliação Estruturada" OR "Marco de evaluación" OR "Marco de valoración" OR "Marcos de evaluación" OR "Marcos de valoración" OR "evaluación estructurada" OR "valoración estructurada" OR "Desarrollo del marco" OR "Cadre d'évaluation" OR "Cadres d'évaluation" OR "Évaluation structurée" OR "Élaboration de cadres" OR "software validation" OR "software validations" OR "software verification" OR "software verifications" OR "Validação de Programas de Computador" OR "Validação de Software" OR "Verificação de Programa de Computador" OR "Verificação de Software" OR "Validación de Programas de Computación" OR "Verificación de Programas de Computación" OR "Validation de logiciel" OR "Vérification de logiciel"))92
MEDLINE/PubMed("Information Systems"[MeSH Terms] OR "information system*"[Title/Abstract] OR "Information Technology"[MeSH Terms] OR "Technology"[mh:noexp] OR "technolog*"[Title/Abstract] OR “Digital Technology”[MeSH Terms] OR Educational Technology[MeSH Terms]) AND ("Evaluation framework*"[Title/Abstract] OR "structured evaluat*"[Title/Abstract] OR "Framework Development"[Title/Abstract] OR "software validat*"[Title/Abstract] OR "software validation"[MeSH Terms] OR “software verificat*"[Title/Abstract])975
ScopusTITLE-ABS-KEY ( "Evaluation framework*" OR "structured evaluat*" OR "Framework* Development*" OR "software validat*" OR "software verificat*" ) AND TITLE-ABS-KEY ( "information system*" OR technolog* )3755
Web of Science(TS="Evaluation framework*" OR TS="structured evaluat*" OR TS="Framework* Development*" OR TS="software validat*" OR TS="software verificat*") AND (TS="information system*" OR TS=technolog*)2004
Google Scholar("Information Systems" OR "Information Technology") AND ("Evaluation framework" OR "software validation")100
Google Scholar("Sistemas de Informação" OR "Sistema Tecnológico") AND ( "Modelo de avaliação" OR "Estrutura de avaliação" OR "Validação de Software")100
Source: Elaborated by the authors (2024).

period were also excluded.

Following the selection process, the included articles were divided between two reviewers for data extraction. Four reviewers collectively created the extraction spreadsheet. A pilot extraction was conducted to calibrate the reviewers before the final extraction. Following the extraction process, the two reviewers cross-checked each other’s extracted data. To conclude, a consensus meeting was held.

The extracted characteristics included the first author, year of publication, country of study, study design, name and/or type of IS, name and/or type of evaluation framework, development stage of the evaluation framework, and available psychometric properties. Lastly, the first and second reviewers analyzed the extracted data, followed by verification by all authors. The results are summarized descriptively in the tables and figures provided herein.

https://doi.org/10.59681/2175-4411.v18.2026.15393
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
Table 2. Eligibility criteria for inclusion in the scoping review.
DomainInclusionExclusion
ProblemCharacteristics, construction stages and/or measurement propertiesNo data on characteristics, construction stages or measurement properties
ConceptEvaluation frameworks1 from the user’s perspectiveMethods not considering the user’s perspective; Methods only addressing the planning and implementation of IS; Studies not presenting the structure of the evaluation framework; Studies not presenting sufficient data on the types of evaluation frameworks
ContextEconomy, agriculture, health, education, environment or infrastructure ISStudies not including IS; Studies only providing data on other technologies
Study typeObservational studies2, clinical trials and systematic reviewsStudies not proposing/presenting an evaluation framework/method; Non-systematic reviews
IS: information systems.
1 Evaluation models, theoretical evaluation frameworks, instruments, questionnaires or any type of tool.
2 Descriptive, cross-sectional and longitudinal.
Source: Elaborated by the authors (2024).

3 Results

Figure 1 presents a summary of the search and selection process. Initially, the searches yielded 18,837 articles, of which 6,011 were excluded as duplicates. Upon reviewing the titles and abstracts, we eliminated an additional 12,780 articles. In the subsequent stage, we assessed 46 articles for potential relevance. However, 34 did not satisfy all inclusion criteria (Table 3). Consequently, 11 studies were included in the review [8-18].

The studies included in this analysis were conducted across various countries, with the United States contributing the most (n = 3); the remaining studies were from Brazil (n = 1) [8], Canada (n = 1) [13], Cyprus (n = 1) [17], India (n = 1) [11], Iran (n = 1) [14], New Zealand (n = 1) [16], Turkey (n = 1) [10], and the United Kingdom (n = 1) [18]. All studies were published in English and from 1999 onward, although most were published after 2006. The primary characteristics of these studies are provided in Table 4.

The studies introduced a range of evaluation frameworks [8-18]. Among the dimensions described in the papers, those assessing quality were the most common, followed by those evaluating user expectations and satisfaction (Figure 2). As for the main characteristics of the frameworks (e.g., methods for application or scoring, etc.) in the sample (Table 5), five reported validating their evaluation frameworks [9,10,15,17,18], four only provided a description of the framework [8,13,14,16], and two developed and applied a framework without validation [11,12]. All studies provided an evaluation framework for IS in healthcare [8-10,12-18], except for one [11], which applied its framework in the healthcare sector but also extended its use to IS in various other fields, such as manufacturing, banking and finance, consulting, commerce,

Figure 1. Flowchart of the article selection process.
Flowchart of the article selection process 2026-11-16T21:39:11.541665 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/
Source: Elaborated by the authors (2024), based on PRISMA 2020.

travel agencies, and education.

The Clinical Information Systems Success Model (CISSM) assesses the effectiveness of clinical information systems across four principal dimensions: system performance, information quality, social influence, and facilitating conditions [9]. The Human, Organization, and Technology-fit (HOT-fit) evaluation framework for HIS integrates human, organizational, and technological factors to assess IS success, underscoring the critical need for alignment among these components [18]. The Development of an Evaluation Framework for Health Information Systems (DIPSA) Framework [17] offers a structured approach to HIS assessment, focusing on performance, organizational impact, and perceived benefits from various stakeholders’ perspectives. Meanwhile, the Electronic Health Record Nurse Satisfaction (EHRNS) framework [15], a questionnaire-based tool, measures nurses’ satisfaction with EHR systems. The Expectation Questionnaire [10] evaluates end users’ expectations of IS to identify discrepancies between expected and actual outcomes.

https://doi.org/10.59681/2175-4411.v18.2026.15394
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
Table 3. Reasons for exclusion of the studies assessed for eligibility.
Reason for exclusionnExcluded studies (first author, year)
1. Non-systematic reviews or systematic reviews without an evaluation method; letters; book chapters; non-peer-reviewed conference papers; opinion articles; articles describing techniques; panels; preprints and clinical/technical guides5Xydopoulos (2013); Christopoulou (2018); Borim (2015); Palvia (2001); Lee (2010)
2. Studies that do not include information systems or that only provide data on another type of technology11Hub (2008); Wang (2014); Lima (2020); Sabau (2009); Cho (2010); Palacio (2010); Shaw (2001); Cresswell (2020); Sockolow (2012); Marthandan (2010); Usener (2012)
3. Evaluation methods that do not consider the user's perspective3Beuscart-Zéphir (1997); Lau (2009); Smith (2011)
4. Studies that do not present the structure of the evaluation framework5Khairata (2019); Stylianides (2019); Heidebrecht (2014); Grant (2002); Baroudi (1988)
5. Studies that do not present the structure of the evaluation framework in the areas of economy, agriculture, health, education, environment and infrastructure7Jiang (1999); Abidin (2019); Andersson (1997); Al-Tunisi (1995); Russo (2018); Helwig (2020); Goodhue (1998)
6. Studies that do not provide enough data about the types of evaluation frameworks for information systems1Kushniruk (2003)
7. Studies that address only the evaluation of the planning and implementation of information systems, without post-implementation evaluation2Boland (2014); Goldberg (2013)
Total34
Source: Elaborated by the authors (2024).
Table 4. Main characteristics of the included studies (n = 11).
StudyParticipant characteristicsValidityReliability
Garcia-Smith (2013) [9] United States. Design: Literature review and methodological study. IS: Clinical Information System. Framework: Clinical Information Systems Success Model (CISSM). Stage: Validated instrument.The population consisted of 234 nurses. The majority were female (91.0%), with an average age of 28 years and 10.4 years of professional experience.Construct validation; Loading factors: System Performance: 0.726-0.839; Information Quality: 0.835-0.882; Social Influence: 0.679-0.841; Facilitating Conditions: 0.713-0.796Internal Consistency; Cronbach's Alpha: System Performance: 0.80; Information Quality: 0.89; Social Influence: 0.85; Facilitating Conditions: 0.89
Gürsel (2014) [10] Türkiye. Design: Methodological study. IS: Hospital Information System. Framework: Expectation Questionnaire. Stage: Validated instrument.There were 504 participants from Hospital A and 96 participants from Hospital B. Among them, there were professionals from different areas of the hospital (administrative, health and others). In both hospitals, the majority had higher education and good experience in information systems.NRInternal Consistency; Cronbach's Alpha: Importance weights: 0.881; Expectation ratings: 0.970
Kanungo (1999) [11] India. Design: Literature review and development of an evaluation framework. IS: Manufacturing, banking and finance, consultancy, trade, travel agencies, education, and hospitals systems. Framework: IS effectiveness model. Stage: Instrument constructed and applied, but not validated.NANANA
Kukhareva (2022) [12] United States. Design: Literature review and development of an evaluation framework. IS: Epic® EHR. Framework: Evaluation in Life Cycle of Information Technology (ELICIT) framework. Stage: Instrument constructed and applied, but not validated.NANANA
Lau (2007) [13] Canada. Design: Systematic review and development of an evaluation framework. IS: Health Information System. Framework: InfowayNANANA
https://doi.org/10.59681/2175-4411.v18.2026.15395
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
StudyParticipant characteristicsValidityReliability
Benefits Evaluation Framework. Stage: Description of an evaluation framework.NANANA
Morais e Costa (2014) [8] Brazil. Design: Literature review and development of an evaluation framework. IS: Information systems of the Unified Health System. Framework: NR. Stage: Description of an evaluation framework.NANANA
Norouzi (2024) [14] Iran. Design: Literature review and development of an evaluation framework. IS: Health information systems. Framework: NR. Stage: Description of an evaluation framework.NANANA
Sockolow (2011) [15] United States. Design: Literature review and methodological study. IS: Electronic Health Records (EHR). Framework: Electronic Health Record Nurse Satisfaction (EHRNS). Stage: Validated instrument.The instrument was assessed for reliability and validity in five phases. In the pilot test, participants were nurses studying for a postgraduate degree with a mean age of 26 years, 5 years of professional experience and 2 years of experience with an EHR (n = 30). In the other four phases, participants (n = 90) had a mean age of 49 years and 51% of respondents were nurses or nurse practitioners at the EHR site, compared with 60% at the paper-based site. Other respondents included therapists, social workers and physicians. In both sites, respondents had approximately 23 years of healthcare experience, almost half had experience with an EHR and, of those with EHR experience, the mean EHR experience was 4 years.Content validity: Items and instructions clear; minor issues with ceiling effect on "availability."; Construct validity supported within and across dimensions. Three item pairs showed significant correlations in both surveys (e.g., usability–overall satisfaction; team communication–patient safety; team communication–patient outcome), with six pairs significant in only one survey. Interviews corroborated four correlated items (support, team communication, overall satisfaction, clinician involvement) and five additional items (available, problems, usability, complete/correct, efficiency). EHR functionality frequently highlighted by cliniciansInternal consistency for EHR: Kendall's tau: 0.35–0.57
Stockdale (2006) [16] New Zealand. Design: Literature review and development of an evaluation framework. IS: NR. Framework: Content, context and process (CCP). Stage: Description of evaluation framework.NANANA
Stylianides (2018) [17] Cyprus. Design: Cross-sectional and methodological study. IS: Health Information Systems. Framework: DIPSA evaluation framework. Stage: Validated instrument.The population consisted of 234 nurses. The majority were female (91.0%), with an average age of 28 years and 10.4 years of professional experience.NRInternal Consistency; Cronbach's Alpha: Technology System quality: 0.913; Safety: 0.855; Human Factor Collaboration: 0.939; Satisfaction: 0.916; Organization Procedures: 0.898
Yusof (2008) [18] United Kingdom. Design: Literature review and development of an evaluation framework. IS: Health Information Systems (HIS). Framework: Human, Organization, and Technology-fit​ (HOT-fit). Stage: Validated instrument.The sample consisted of 15 health professionals associated with primary care and hospitals.NRNR
IS: information system. Stage: stage of the framework’s construction. NA: not applicable. NR: not reported.
Source: Elaborated by the authors (2024).
https://doi.org/10.59681/2175-4411.v18.2026.15396
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539

The IS Effectiveness Model [11] is a more generic framework for evaluating IS effectiveness by analyzing technical quality, user satisfaction, and the impact on organizational goals. This approach incorporates both internal evaluations (by CEOs, IS managers, and users) and external assessments (by IT experts), culminating in a composite score derived from a formula that integrates these perspectives. The Content, Context, and Process evaluation framework focuses on health interventions [16], examining intervention content, implementation context, and the processes executed to achieve outcomes. Despite its detailed theoretical basis, this framework has not been applied or validated in practice. Likewise, the Infoway Benefits Evaluation Framework targets HIS [13], emphasizing accessibility, quality of care, and operational efficiency, although it also remains unvalidated. The Evaluation in Life Cycle of Information Technology [12], designed to assess IS throughout their lifecycle with a particular focus on the EPIC® EHR system in the United States, aims to evaluate systems from planning through implementation, emphasizing the fulfillment of organizational needs. Moreover, two studies introduced frameworks without designated titles. Morais and Costa [8] presented a framework for evaluating the IS of the Brazilian Unified Health System, identifying critical indicators of product quality. In Iran, Norouzi et al. [14] developed a framework based on literature from 2000 to 2018, proposing indicators related to system performance, organizational aspects, system design, and technology.

What is more, among the surveyed studies (Table 4), only five reported validating their evaluation frameworks [9,10,15,17,18]. However, the study utilizing the HOT-fit framework did not disclose its validation results [18]. While all five studies provided details on participant sample size and profession [9,10,15,17,18], only three included information on participants’ sex, age, and professional experience [9,15,17].

The CISSM framework underwent validation with 234 nurses across four hospitals [9]. Construct validity was assessed through factor analysis, which revealed that items loaded onto single factors for each dimension. System Performance explained 63% of the variance, thus supporting Hypothesis 1, while Information Quality, which considers usefulness, completeness, format, and accuracy, accounted for 75% of the variance, validating Hypothesis 2. Social Influence, characterized by social factors, services, and support, constituted 55% of the variance, confirming Hypothesis 3. Facilitating Conditions, encompassing behavioral control and work processes, represented 56% of the variance, supporting Hypothesis 4. During the integration stage, Information Quality and Social Influence emerged as the primary predictors of System

Figure 2. Frequency of dimensions in the evaluation frameworks.
Frequency of dimensions in evaluation frameworks 2026-11-16T21:39:11.643984 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/
Source: Elaborated by the authors (2024).

Use Dependency, explaining 25% of its variance. Nurse Satisfaction was influenced by Information Quality, System Performance, and Facilitating Conditions, which collectively accounted for 70% of the variance, with Information Quality acting as the largest contributing factor. At the outcome stage, Nurse Satisfaction predicted 54% of the variance in net benefits, while System Use Dependency accounted for 14%. Greater system integration was associated with increased dependency, satisfaction, and perceived benefits across the hospitals. Reliability was assessed using Cronbach’s alpha, yielding values of 0.80 (system performance), 0.89 (information quality), 0.85 (social influence), and 0.89 (facilitating conditions).

The DIPSA Framework [17] was validated in a study involving 309 healthcare professionals across two public hospitals in Cyprus. Construct validity was assessed through factor analysis, simplifying the original 42 questions across seven categories into 27 questions across five categories. Although the initial factor analysis revealed 10 potential categories, only five met the inclusion criteria, which required each category to retain at least five questions with a factor loading of ≥0.41. These final categories included satisfaction, system quality, collaboration, procedures, and safety. Internal consistency was evaluated using Cronbach’s alpha, yielding the following results: system quality (0.913) and safety (0.855) under the technology factor; collaboration (0.939) and satisfaction (0.916) under the human factor, and procedures (0.898) under the organization factor.

https://doi.org/10.59681/2175-4411.v18.2026.15397
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
Table 5. Main characteristics of the evaluation frameworks.
Study, evaluation framework and dimensionsItemsOperationalization: total number of items, items per dimension and scoring
Garcia-Smith (2013) [9] Clinical Information Systems Success Model (CISSM). Dimensions: System performance; Information quality; Social influence; Facilitating conditions; CIS use dependency; Nurse satisfaction; Net benefit.A. System Performance: 1. Perceived Ease of Use; 2. Accessibility; 3. Reliability; B. Information Quality; 4. Perceived Usefulness; 5. Content Completeness; 6. Format; 7. Accuracy; C. Social Influence; 8. Social Factors; 9. Service Support; D. Facilitating Conditions; 10. Perceived Behavioral Control; 11. Work Processes; E. CIS Use Dependency; F. Nurse Satisfaction; G. Net BenefitFour dimensions. 26 items and 6-point Likert response; scale anchored by “very strongly disagree and “very strongly agree.”
Gürsel (2014) [10] Expectation Questionnaire. Dimensions: Usage expectations; System and data expectations; Improvement expectations; Managerial expectations.A. Usage expectations; 1. Ease-of-use; 2. Need for training; 3. Help manuals; 4. Speed; 5. User; B. System and data expectations; 6. Consistency; 7. Privacy; 8. Security; 9. Availability; 10. Interoperability; C. Improvement expectations; 11. Improve service quality; 12. Decreasing workload; 13. Bringing positive change; 14. Research facilities support; D. Managerial expectations; 15. Reporting facilities; 16. Decision support; 17. Function SufficiencyUsers are asked to express their expectation rating using the; 5-point Likert scale (strongly agree, moderately agree, not sure, moderately disagree, and strongly disagree); and importance weights using the 5-point Likert scale (very important, important, averagely important, not so; important, and not important), which are our linguistic variables in fuzzy logic operations.; Importance Weights: Each level of importance has an associated fuzzy number: Very important (W₁): Represented by the fuzzy number (0.75, 1, 1).; Important (W₂): Represented by (0.5, 0.75, 1).; Averagely important (W₃): Represented by (0.25, 0.5, 0.75).; Not so important (W₄): Represented by (0, 0.25, 0.5).; Not important (W₅): Represented by (0, 0, 0.25).; Expectation Ratings: The expectation ratings also have corresponding fuzzy numbers: Strongly agree (R₁): Represented by (0.5, 1, 1).; Moderately agree (R₂): Represented by (0, 0.5, 1).; Not sure (R₃): Represented by (-0.5, 0, 0.5).; Moderately disagree (R₄): Represented by (-1, -0.5, 0).
Kanungo (1999) [11] IS effectiveness model. Dimensions: Effectiveness.1. Predicting future trends; 2. Improving decision making; 3. Avoiding problem areas; 4. Increased user satisfaction; 5. Improving systems integration; 6. Improving resource allocation; 7. Improving speed of operations; 8. Facilitating information retrieval; 9. Improving control over systems; 10. Minimizing errors/mistakes in functional areas; 11. Improving systems standardization; 12. Reducing response time to changes in requirements; 13. Improving quality of product/service; 14. Facilitating communications with customers/suppliersRespondents at each level in each organization; were asked to score each factor in Table 4 on a; 10-point scale (10 being most important and 0; being not relevant). In addition, the research; team also organized a one-day workshop of IT; managers and consultants who provided an; independent assessment of the different factors; based on his or her observations. Thus, each; organization has three internal (CEO, IS manager and user) and one external; evaluations. The final evaluation was obtained; by using the following formula: Score final = (score internal + score external)/2 "where score internal = score CEO + Score IS manager + Score User)/3"
Kukhareva (2022) [12] Evaluation in Life Cycle of Information Technology (ELICIT) framework. Dimensions: Society, user, and IT.Measure Levels; Society; Implementation outcomes: System reach; System adoption; Process outcomes: Guideline compliance; Process efficiency; Health outcomes: Patient health; Equity; Safety; Financial value: Cost; Cost-effectiveness; Return on investment; User; Usability: Ease of use; Ease of understanding; Ease of learning; Workflow: Workflow fragmentation; Number of screens; Cognitive load; System use: Amount/duration; Number of functions used; Number of records assessed; User satisfaction: Perceived usefulness; Enjoyment; Trust in the system; IT; System quality: Performance; Functionality; Security; Information quality: Importance; Relevance; Accuracy; Service quality: Responsiveness; Assurance; Follow-up; Interoperability: Portability; Standards complianceOperation Phase; Technical portability assessment; Definition: Determines the portability of the software across different health systems and EHR platforms.; Methods: Evaluation of interoperability standards (e.g., FHIR), assessment of resources needed for external implementations.; Long-term user satisfaction assessment; Definition: Measures continued user acceptance, usability, and the system's perceived impact on care quality.; Methods: System Usability Scale (SUS) surveys, interviews, and Net Promoter Score (NPS) analysis.; Social outcomes assessment; Definition: Assesses process outcomes, clinical impacts, unintended consequences, health equity, economic outcomes, and the dissemination value of the innovation.; Methods: Clinical trial designs (e.g., before-after, interrupted time series), EHR log analysis, cost-effectiveness analysis, and interviews about unintended outcomes.
Lau (2007) [13] Infoway Benefits Evaluation Framework. Dimensions: System quality; Information quality; Service quality; System use; User satisfaction; Net benefits.A. System quality: 1. Functionality; 2. Performance; 3. Security; B. Information quality: 4. Content; 5. Availability; C. Service quality: 6. Responsiveness; D. System Use: 7. Use behavior/pattern; 8. Self-reported use; 9. Intention to use; E. User satisfaction: 10. Competency; 11. User satisfaction; 12. Ease of use; F. Net benefits: 13. Quality: patient safety; appropriateness/effectiveness; health outcomes.; 14. Access: Ability of patients/providers to access services; patient and caregiver participation.; 15: Productivity: efficiency; care coordination; net cost.NR
https://doi.org/10.59681/2175-4411.v18.2026.15398
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
Study, evaluation framework and dimensionsItemsOperationalization: total number of items, items per dimension and scoring
Morais e Costa (2014) [8] NR. Dimensions: Product quality.A. Product quality; 1. Performance efficiency; 2. Usability; 3. Security; 4. Maintainability; 5. PortabilityNR
Norouzi (2024) [14] NR. Dimensions: System performance; Organization; System design; Technology.System Performance; Criteria: Efficiency; Effectiveness; Compatibility; Data Security; Quality and Safety; 2. Organization; Criteria: Attitude; Management; Cost; Workflow; 3. System Design; Criteria: Usability; 4. Technology; Criteria: Database Design; Software; Hardware; Communications; System Security; Software DesignIt as a comprehensive hierarchical set of evaluation indicators for assessing health information systems, developed using the Analytical Network Process (ANP) method. This model includes: 4 dimensions,; 16 criteria,; 47 markers,; 180 indicators.
Sockolow (2011) [15] Electronic Health Record Nurse Satisfaction [EHRNS]. Dimensions: Structural quality; Quality of information logistics; Effects on quality of processes; Effects on outcomes and quality of care; Unintended consequences or benefits; Barriers or facilitators to clinicians’ adoption.A. Structural quality; 1. The patient health record is consistently available; 2. The patient health record is subject to frequent system problems*; 3. The patient health record is user-friendly; 4. Sufficient support is available to operate the patient’s health record; 20. Sufficient resources are provided for me to learn to use the patient health record; B. Quality of information logistics; 5. Patient care data being recorded are accurate and valid; 6. Patients have concerns about security or confidentiality*; 13. The patient health record is worth the time and effort required to use it; 14. Overall, I am satisfied with the patient health record; 15. I think patients are satisfied with clinician use of the patient health record; C. Effects on quality of processes; 7. Patient care services are provided in a timely manner; 8. Patient care orders are appropriate; 10. Supports effective communication between most team members about patient care; 16. My department had a role in the introduction of the patient health record at my facility; 17. People who use the patient health record should have had more to say about its design*; D. Effects on outcomes and quality of care; 9. Contributes to the safety of patients; 11. Contributes to patient outcomes; 12. Contributes to patient’s knowledge of their health condition; 21. Part of the increase in costs of healthcare is because of computers*; E. Unintended consequences/benefits; 18. I have firsthand knowledge that problems with the patient health record have interfered with patient care*; F. Barriers/facilitators adoption; 19. My facility is more likely to adopt a computerized patient health record system if the system could exchange patient; information with nursing homes and hospitalsThere are 21 questions divided into six dimensions. The questions are scored using a 5-point Likert scale ranging from 0 = strongly disagree (most dissatisfied) to 5 = strongly agree (most satisfied). Five items (marked with *) were formulated negatively.
Stockdale (2006) [16] Content, context and process (CCP). Dimensions: Content; Context; Process.NANR
Stylianides (2018) [17] DIPSA evaluation framework; D: "Διαδικασίες" (Procedures); I: "Ικανοποίηση" (Satisfaction); P: "Ποιότητα" (System Quality); S: "Συνεργασία" (Collaboration); A: "Ασφάλεια" (Safety). Dimensions: Satisfaction, collaboration,; safety, system quality, procedures.Human Factor: Satisfaction: Measures if users of IHIS are satisfied with the system, considering effort, quality, and performance.; Collaboration: Evaluates whether the system supports collaboration among healthcare professionals.; Technology: System Quality: Assesses aspects such as availability, reliability, access, and the quality of information provided by the system.; Safety: Focuses on how the system enhances patient safety and prevents errors.; Organization: Procedures: Examines the impact of the system on the daily procedures of healthcare professionals.Total Number of Items; The framework includes 42 questions in the initial questionnaire, with 27 questions ultimately grouped into the five main categories (dimensions) based on factor analysis.; 2. Items per Dimension; The five dimensions consist of the following number of items: Satisfaction: 5 items; Collaboration: 5 items; System Quality: 6 items; Safety: 6 items; Procedures: 5 items; 3. Scoring; The questionnaire uses a Likert scale ranging from 1 (lowest value) to 5 (highest value) for most of the items.; Scores for each dimension are calculated based on the responses to the items within that category.
Yusof (2008) [18] Human, Organization, and Technology-fit​ (HOT-fit). Dimensions: Technology; Human; Organization; satisfaction, collaboration,; safety, system quality, procedures.Technology: System Quality; Information Quality; Service Quality; Human: System Use; User Satisfaction; Organization: Organizational Structure; Organizational Environment; Net Benefits: The overall impacts of the system on individual users, organizations, and healthcare deliveryTechnology; System Quality: Data accuracy, data currency, database contents, ease of use, ease of learning, availability, usefulness of system features and functions, flexibility, reliability, technical support, security, efficiency, resource utilization, response time, turnaround time.; Information Quality: Importance, relevance, usefulness, legibility, format, accuracy, conciseness, completeness, reliability, timeliness, data entry methods.; Service Quality: Quick responsiveness, assurance, empathy, follow-up service, technical support.; Human; System Use: Amount/duration (e.g., number of inquiries, amount of connect time, number of functions
https://doi.org/10.59681/2175-4411.v18.2026.15399
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
Study, evaluation framework and dimensionsItemsOperationalization: total number of items, items per dimension and scoring
Human, Organization, and Technology-fit​ (HOT-fit). Dimensions: Technology; Human; Organization; satisfaction, collaboration,; safety, system quality, procedures.Technology: System Quality; Information Quality; Service Quality; Human: System Use; User Satisfaction; Organization: Organizational Structure; Organizational Environment; Net Benefits: The overall impacts of the system on individual users, organizations, and healthcare deliveryused, number of records accessed, frequency of access, frequency of report requests, number of reports generated), use by whom? (direct vs. chauffeured use), actual vs. reported use, nature of use (use for intended purpose, appropriate use, type of information used), purpose of use, level of use (general vs. specific), recurring use, report acceptance, percentage used, voluntariness of use, motivation to use, attitude, expectations/belief, knowledge/ expertise, acceptance, resistance/reluctance, training.; User Satisfaction: Satisfaction with specific functions, overall satisfaction, perceived usefulness, enjoyment, software satisfaction, decision-making satisfaction.; Organization; Structure: Nature (type, size), culture, planning, strategy, management, clinical process, autonomy, communication, leadership, top management support, medical sponsorship, champion, mediator, teamwork.; Environment: Financing source, government, politics, localization, competition, inter-organizational relationship, population served, external communication.; Net Benefits; Clinical Practice: (Job effects, task performance, productivity, work volume, morale), efficiency, effectiveness (goal achievement, service), decision-making quality (analysis, accuracy, time, confidence, participation), error reduction, communication, clinical outcomes (patient care, morbidity, mortality), cost.
NA: not applicable. NR: not reported.
Source: Elaborated by the authors (2024).

The EHRNS framework [15] was validated in the context of EHR implementation by surveying 37 participants at 11 months post-implementation and 32 participants at 17 months post-implementation. The instrument was deemed clear with relevant items; however, the ‘availability’ item demonstrated a ceiling effect in both samples, indicating a need for revision. Construct validity was partially supported, with significant correlations observed between three item pairs across dimensions (e.g., ‘team communication’ with ‘patient safety’ and ‘patient outcome’), suggesting that the framework would benefit from further validation. Qualitative observations and interviews corroborated the validity of items such as ‘support’ and ‘overall satisfaction.’ Internal consistency, assessed using Kendall’s tau, revealed significant correlations in three of the five related item pairs (e.g., ‘support-resources’ and ‘overall satisfaction-worthwhile’), indicating moderate reliability. The variability in responses reflected diverse user perceptions of satisfaction.

The Expectation Questionnaire [10] was validated by analyzing the importance weights and expectation ratings provided by hospital IS users across two hospitals in Turkey. The validation involved a sample of 600 participants (504 from Hospital A and 96 from Hospital B), with response rates of 76.4% and 96.0%, respectively. The importance weights reflect the relevance users assign to each evaluated variable, measured on a 5-point Likert scale ranging from ‘not important’ to ‘very important.’ Conversely, expectation ratings gauge the extent to which user expectations are met, also evaluated on a 5-point Likert scale from

‘strongly disagree’ to ‘strongly agree.’ To facilitate more nuanced analyses and precise computations of the expectation-meeting index, these data were transformed into fuzzy triangular numbers. The internal consistency of these metrics was assessed using Cronbach’s alpha. For importance weights, the coefficients were 0.871 for Hospital A and 0.942 for Hospital B. For expectation ratings, the scores were 0.966 for Hospital A and 0.959 for Hospital B. The overall Cronbach’s alpha values for these two dimensions were 0.881 and 0.970, respectively.

4 Discussion

This review identified 11 evaluation frameworks for IS from the users’ perspective, mainly within the health domain. The frameworks showed diverse characteristics, underscoring the complexity and multidimensionality of IS. Differences in focus and scope reflect varied methodological approaches, demonstrating adaptability to different contexts. However, their practical utility is frequently limited by a lack of rigorous validation. Among the few studies that reported measurement properties, empirical assessments were often incomplete, thereby reducing both the generalizability of the frameworks and confidence in their findings.

Most studies focused on quality, satisfaction, and user expectations, though they differed in their specific approaches. A common feature was the emphasis on multiple dimensions (e.g., technical quality, user satisfaction, and organizational impact), highlighting the multifaceted nature of IS. Frameworks such as CISSM [9] and HOT-fit [18] stressed the alignment of

https://doi.org/10.59681/2175-4411.v18.2026.153910
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539

human, organizational, and technological factors, recognizing that IS effectiveness depends on the continuous interaction among these elements. Other frameworks, including the IS Effectiveness Model [11] and DIPSA [17], assessed system impact on organiza‐tional goals and processes to ensure strategic and operational alignment. Notably, DIPSA [17] explicitly builds upon the core variables established in HOT-fit [18].

The evaluated frameworks also differ significantly in scope. DIPSA prioritizes performance and stakeholder benefits, whereas the Infoway Benefits Evaluation Framework [13] focuses on accessibility and quality of care. The Content, Context, and Process framework [16] specifically analyzes health interventions, whereas the IS Effectiveness Model [11] adopts a broader perspective applicable outside the healthcare sector. These variations show that although all the frameworks aim to assess IS, their methods and scopes are tailored to address distinct organizational needs.

Only five studies validated their frameworks [9,10,15,17,18], and one of these failed to report its empirical results. Despite showing promise in terms of reliability and construct validity, the validation efforts in these studies often lacked methodological depth and breadth. Although all used established methods (e.g., factor analysis), the underlying rationale for initial dimension definition was often unclear. Furthermore, factor analysis alone does not ensure robust validation, especially given the requirement for ongoing system adaptation [19].

Content validation was similarly insufficient, typically relying only on expert input without involving end users or accounting for cultural adaptation. This oversight weakens generalizability and undermines confidence in the overarching conclusions, underscoring the need for rigorous empirical validation to ensure accuracy across diverse contexts. The findings also stress the inclusion of core dimensions such as usability, satisfaction, information quality, and security, which collectively reflect both technical efficiency and successful integration into organizational and user environments [20]. While IS adoption is often studied through the lens of specific barriers and facilitators, advancing the theoretical understanding of system adoption, dissemination, and long-term maintenance remains essential. Evaluation models have thus evolved to prioritize key determinants such as perceived usefulness and ease of use [21].

The included studies covered a wide geographic range, and all were published in English. A significant portion of the research was conducted in the United States [9, 12,15], with the remaining studies being in Brazil, Canada, Cyprus, Iran, New Zealand, Turkey, and India. This distribution highlights a persistent disparity in the

geographic representation of user-centered evaluations for HIS [22].

Despite employing broad search strategies, all included studies ultimately focused on HIS [8-18]. This dominance suggests a gap in the development of user-centered evaluation frameworks within other IS domains. Although one analyzed framework was structurally generic, most addressed specific healthcare contexts, indicating a clear need to extend these evaluative applications to other sectors, such as education and public administration.

Furthermore, despite core dimensions (e.g., interoperability, standards adoption, semantic consistency, and legacy data integration) being widely recognized as critical for the performance and scalability of HIS, these technical aspects are rarely captured by evaluations grounded in the end-user perspective. User-centered assessments tend to focus on attributes directly experienced during daily system interaction, including quality, functionality, usability, and perceived usefulness.

Technical and architectural characteristics (e.g., interoperability or semantic alignment) typically require specialized analyses conducted by system developers, IT managers, or interoperability experts; thus, they fall outside the direct evaluative capacity of most end users. Viewed from this perspective, the predominance of generalized evaluation dimensions identified in this review does not necessarily reflect a limitation of the primary studies, but rather an intrinsic characteristic of user-based evaluation methods.

This scoping review is subject to certain limitations. Extensive methodological variability across studies hindered direct comparability and analytical consistency. Differences in validity assessments and a lack of detailed data (e.g., missing item descriptions, incomplete framework operationalization, and unpublished validation results) further reduced transparency and replicability. Additionally, publication bias may have influenced the synthesized findings. Nonetheless, the review’s strengths include a comprehensive scoping search strategy and the timely relevance of the topic given the accelerating global adoption of health information technologies.

5 Conclusion

This scoping review provides a comprehensive analysis of IS evaluation models, particularly within the healthcare sector. It highlights the prevailing methodological diversity, the limited generalizability of existing tools, and the absence of standardized evaluation dimensions. Although the included studies cover varied approaches and contexts, thereby enhancing our understanding of current evaluation practices, significant challenges remain in result

https://doi.org/10.59681/2175-4411.v18.2026.153911
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539

comparability, analytical transparency, and generalizability. These limitations emphasize the urgent need for further research utilizing standardized methodologies, broader demographic populations, and longitudinal study designs. Nevertheless, this review notes important strengths in the literature, including the high clinical relevance of the evaluated topics and the broad range of dimensions addressed, advancing knowledge on the reliability and validity of health evaluation models.

Future research should focus on refining specific evaluation models and would benefit from integrating user-centered evaluation approaches with

complementary technical or organizational assessments, such as those related to system integration or interoperability, to provide a more comprehensive understanding of HIS and their downstream implications for healthcare management and service quality.

Lastly, bridging the gap between evaluation frameworks developed in academic research and their practical adoption by the organizations that develop or deploy HIS may contribute to more systematic assessments, improve end-user satisfaction, and enhance overall productivity in real-world clinical settings.

Author contributions. The authors contributed equally to all stages of the article: conception and design of the study; data collection, analysis, and interpretation; drafting and critical revision of the manuscript; and approval of the final version.
Acknowledgments. The authors would like to thank Karyn Munik Lehmkuhl for her support with the search strategies and Atlas Assessoria Linguística for language editing.
Funding. The authors did not declare any source of funding.
Data and materials availability. The authors did not provide a data and materials availability statement.
Conflicts of interest. The authors did not declare any conflicts of interest.
Use of artificial intelligence. The authors did not declare the use of artificial intelligence tools.

References

  1. Pleskach V, Pleskach M, Zelikovska O. Information security management system in distributed information systems. In: 2019 IEEE International Conference on Advanced Trends in Information Theory (ATIT); 2019. p. 300-3. doi: https://doi.org/10.1109/ATIT49449.2019.9030484
  2. Gregersen E. 5 components of information systems. Encyclopaedia Britannica; 2021.
  3. Stoykova S, Shakev N. Artificial intelligence for management information systems: opportunities, challenges, and future directions. Algorithms. 2023;16(8):357. doi: https://doi.org/10.3390/a16080357
  4. Maia MAQ, Barbosa RR, Williams P. Usabilidade e experiência do usuário de sistemas de informação: em busca de limites e relações. Ciênc Inf Rev. 2020;6(3):34-48. doi: https://doi.org/10.28998/cirev.2019v6n3c
  5. BSI. ISO 9241-210:2019. Ergonomics of human-system interaction. Part 210: Human-centred design for interactive systems. London: BSI; 2019.
  6. Cintho LM, Machado RR, Moro CMC. Métodos para avaliação de sistema de informação em saúde. J Health Inform. 2016;8(2). Disponível em: https://jhi.sbis.org.br/index.php/jhi-sbis/article/view/346
  7. Osborne SE. Evaluating health care information systems: methods and applications. J Adv Nurs. 1994;20(1):203-4. doi: https://doi.org/10.1046/j.1365-2648.1994.20010202-6.x
  8. Morais RM, Costa AL. Um modelo para avaliação de sistemas de informação do SUS de abrangência nacional: o processo de seleção e estruturação de indicadores. Rev Adm Pública. 2014;48(3):767-93. doi: https://doi.org/10.1590/0034-76121512
  9. Garcia-Smith D, Effken JA. Development and initial evaluation of the Clinical Information Systems Success Model (CISSM). Int J Med Inform. 2013;82(6):539-52. doi: https://doi.org/10.1016/j.ijmedinf.2013.01.011
  10. Gürsel G, Zayim N, Gülkesen KH, Arifoğlu A, Saka O. A new approach in the evaluation of hospital information systems. Turk J Elec Eng Comp Sci. 2014;22(1):214-22. doi: https://doi.org/10.3906/elk-1110-15
  11. Kanungo S, Duda S, Srinivas Y. A structured model for evaluating information systems effectiveness. Syst Res Behav Sci. 1999;16(6):495-518. doi: https://doi.org/10.1002/(SICI)1099-1743(199911/12)16:6<495::AID-SRES238>3.0.CO;2-R
  12. Kukhareva PV, Weir C, Del Fiol G, Aarons GA, Taft TY, Schlechter CR, et al. Evaluation in Life Cycle of Information Technology (ELICIT) framework: supporting the innovation life cycle from business case assessment to summative evaluation. J Biomed Inform. 2022;127:104014. doi: https://doi.org/10.1016/j.jbi.2022.104014
  13. Lau F, Hagens S, Muttitt S. A proposed benefits evaluation framework for health information systems in Canada. Healthc Q. 2007;10(1):112-6, 118.
  14. Norouzi H, Mehrolhassani MH, Hajesmaeel-Gohari S, Ahmadian L, Ghaemi MM, Mohammadi M, et al. Identification and classification of indicators for evaluating health information systems. Health Policy Technol. 2024;13(2):100833. doi: https://doi.org/10.1016/j.hlpt.2023.100833
  15. Sockolow PS, Weiner JP, Bowles KH, Lehmann HP. A new instrument for measuring clinician satisfaction with electronic health records. Comput Inform Nurs. 2011;29(10):574-85. doi: https://doi.org/10.1097/NCN.0b013e31821a1568
https://doi.org/10.59681/2175-4411.v18.2026.153912
Rampinelli VPC et al.J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539

  1. Stockdale R, Standing C. An interpretive approach to evaluating information systems: a content, context, process framework. Eur J Oper Res. 2006;173(3):1090-102. doi: https://doi.org/10.1016/j.ejor.2005.07.006
  2. Stylianides A, Mantas J, Roupa Z, Yamasaki EN. Development of an evaluation framework for health information systems (DIPSA). Acta Inform Med. 2018;26(4):230-4. doi: https://doi.org/10.5455/aim.2018.26.230-234
  3. Yusof MM, Kuljis J, Papazafeiropoulou A, Stergioulas LK. An evaluation framework for health information systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform. 2008;77(6):386-98. doi: https://doi.org/10.1016/j.ijmedinf.2007.08.011
  4. Reichenheim M, Bastos JL. What, what for and how? Developing measurement instruments in epidemiology. Rev Saude Publica. 2021;55:40. doi: https://doi.org/10.11606/s1518-8787.2021055002813
  5. Barzekar H, Ebrahimzadeh F, Luo J, Karami M, Robati Z, Goodarzi P. Adoption of hospital information system among nurses: a technology acceptance model approach. Acta Inform Med. 2019;27(5):305-10. doi: https://doi.org/10.5455/aim.2019.27.305-310
  6. Ammenwerth E. Technology acceptance models in health informatics: TAM and UTAUT. Stud Health Technol Inform. 2019;263:64-71. doi: https://doi.org/10.3233/SHTI190111
  7. Rana RH, Alam K, Gow J. Health outcome and expenditure in low-income countries: does increasing diffusion of information and communication technology matter? Inf Technol Dev. 2020;26(3):506-24. doi: https://doi.org/10.1080/02681102.2019.1678455

Editorial notes

The Journal of Health Informatics is a scientific journal maintained by the Brazilian Society of Health Informatics (Sociedade Brasileira de Informática em Saúde), a non-profit scientific association dedicated to the development of Health Informatics and Digital Health in Brazil. The editorial process is conducted with scientific independence, free from institutional, commercial, or sponsor interference, and follows international best practices in publication ethics.

This article was double-blind peer reviewed by specialists in the field. The editorial team used artificial intelligence tools, under human supervision, in the initial screening, in checking the text, references and numbers and, when needed, in redrawing figures, charts and tables, which the authors approved in the proof. The tools and safeguards are described in the journal's Policy on the use of artificial intelligence. Editorial decisions were made by people.

https://doi.org/10.59681/2175-4411.v18.2026.153913
Rampinelli VPC et al.
J Health Inform. 2026;18:e1539
https://doi.org/10.59681/2175-4411.v18.2026.1539
J Health Inform. 2026;18:e1539
DOI: https://doi.org/10.59681/2175-4411.v18.2026.1539
Review Article

An evaluation framework for information systems from the users’ perspective: a scoping review

Estrutura de avaliação para sistemas de informação na perspectiva dos usuários: uma revisão de escopo
Un marco de evaluación para sistemas de información desde la perspectiva de los usuarios: una revisión de alcance
Vanessa Pereira Corrêa Rampinelli1iD, Catiele Raquel Schmidt1iD, Júlia Meller Dias de Oliveira1iD, Ianka Cristina Celuppi1iD, Arthur Thives Mello1iD, Jades Fernando Hammes2iD, Célio Luiz Cunha2iD, Raul Sidnei Wazlawick1iD, Danielle Soares Rocha Vieira1iD, Eduardo Monguilhott Dalmarco1iD
1 Federal University of Santa Catarina (UFSC), Florianópolis (SC), Brazil.
2 Laboratório Bridge, Florianópolis (SC), Brazil.

Article information

Type Review Article
Identifier e1539
DOI 10.59681/2175-4411.v18.2026.1539
Received 8 September 2025
Accepted 16 April 2026
Published20 May 2026
Review process Double Blind Review
Handling editorJuliano de Souza Gaspar
Access Open, CC BY 4.0
Corresponding authorVanessa Pereira Corrêa Rampinelli
fisio.vpc@gmail.com

License

This is an open access article published under the Creative Commons CC BY 4.0 license, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited.
CC BY 4.0
Abstract
This scoping review aimed to synthesize scientific literature on information systems evaluation methods from the users’ perspective, focusing on their characteristics, construction stages, and measurement properties. The search was conducted in 2024 across the following databases: ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, LILACS, MEDLINE (via PubMed), Scopus, and Web of Science. The study selection process was carried out independently by two reviewers. A total of 18,837 articles were identified, and 11 were ultimately included in the review. The studies presented various evaluation frameworks specific to health information systems, with one study also evaluating information systems from other fields. Only five studies reported the frameworks' validation procedures, and very few provided comprehensive validation results or sufficiently thorough analyses. This scoping review underscores the methodological diversity in information systems evaluation models and highlights the limitations of current use. It emphasizes the need for standardized approaches to enhance comparability, transparency, and long-term system development.
Keywords: Evaluation models; Information systems; Information technology.
Resumo
Esta revisão de escopo teve como objetivo sintetizar a literatura científica sobre estruturas de avaliação de sistemas de informação sob a perspectiva dos usuários, com foco em suas características, etapas de construção e propriedades de mensuração. A busca foi realizada em 2024 nas seguintes bases de dados: ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, LILACS, MEDLINE (via PubMed), Scopus e Web of Science. O processo de seleção dos estudos foi realizado de forma independente por dois revisores. Um total de 18.837 artigos foram identificados nas buscas, e 11 foram incluídos na revisão. Os estudos apresentaram diversas estruturas de avaliação específicas para sistemas de informação em saúde, com um estudo também avaliando sistemas de informação de outras áreas. Apenas cinco estudos relataram os procedimentos de validação das estruturas, e poucos forneceram resultados abrangentes das validações ou análises suficientemente completas. Esta revisão de escopo destaca a diversidade metodológica nos modelos de avaliação de sistemas de informação e destaca as limitações do uso atual. Enfatiza a necessidade de abordagens padronizadas para aprimorar a comparabilidade, a transparência e o desenvolvimento de sistemas a longo prazo.
Descritores: Modelos de avaliação; Sistemas de informação; Tecnologia da informação.
Resumen
Esta revisión de alcance busca sintetizar la literatura científica sobre métodos de evaluación de sistemas de información desde la perspectiva de los usuarios, centrándose en sus características, etapas de construcción y propiedades de medición. La búsqueda se realizó en 2024 en las siguientes bases de datos: ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, LILACS, MEDLINE (vía PubMed), Scopus y Web of Science. El proceso de selección de estudios fue realizado de forma independiente por dos revisores. Se identificaron 18.837 artículos en las búsquedas, de los cuales 11 se incluyeron en la revisión. Los estudios presentaron diversos marcos de evaluación específicos para sistemas de información en salud, y un estudio también evaluó sistemas de información de otros campos. Solo cinco estudios informaron sobre los procedimientos de validación de los marcos, y muy pocos proporcionaron resultados completos de las validaciones o análisis suficientemente exhaustivos. Esta revisión de alcance subraya la diversidad metodológica de los modelos de evaluación de sistemas de información y destaca las limitaciones de su uso actual. Enfatiza la necesidad de enfoques estandarizados para mejorar la comparabilidad, la transparencia y el desarrollo de sistemas a largo plazo.
Descriptores: Modelos de evaluación; Sistemas de información; Tecnología de la información.
HOW TO CITE THIS ARTICLE: Rampinelli VPC, Schmidt CR, Oliveira JMD, Celuppi IC, Mello AT, Hammes JF, et al. An evaluation framework for information systems from the users’ perspective: a scoping review. J Health Inform 2026;18:e1539. doi: https://doi.org/10.59681/2175-4411.v18.2026.1539

1 Introduction

Information systems (IS) are organized sets of components designed to collect, process, store, and distribute information. They support decision-making, coordination, control, analysis, and visualization within organizations [1]. These components include hardware, software, networks, data, and personnel, which work together to streamline operations and deliver relevant information [2]. Health information systems (HIS) are well established within this domain, and IS also play key roles in public sectors such as education, mobility, security, and geolocation.

Evaluating the quality of IS is complex due to the multitude of factors involved. Deploying these systems across various sectors introduces further challenges in aligning multidimensional quality aspects with specific user needs. Understanding both their advantages and challenges, especially from the user’s perspective, is essential [3]. IS use refers to how individuals apply computer systems in their professional activities, reflecting both adoption and effectiveness.

Users, whether individuals, groups, or organizations, interact with these systems to perform tasks, access information, conduct transactions, or make decisions. They can be categorized as end, administrative, technical, managerial, or external users [4]. End users rely on the system for daily tasks; examples include healthcare professionals using electronic health records (EHR) or employees accessing enterprise management systems. Defining user categories is essential for designing, implementing, and evaluating systems, as each group’s needs and expectations influence usability, efficiency, and overall success [5].

Despite the importance of user-centered evaluation, few studies present comprehensive frameworks from this perspective. Most available studies come from the health sector or are over thirty years old [6,7]. Many focus on technical or organizational aspects rather than the user experience, prioritizing performance, implementation, or economic impacts over acceptance, satisfaction, and ease of use. Evaluations also tend to be system-specific, limiting generalizability. Thus, a significant gap remains regarding frameworks that incorporate the user perspective.

This manuscript examines key sectors of national governance: the economy, agriculture, health, education, the environment, and infrastructure, which are essential for resource management, policy formulation, and addressing societal needs equitably and sustainably. IS in these areas support data collection, analysis, and dissemination, thereby improving decision-making, transparency, and efficiency. This study addresses the current gap in the literature by synthesizing IS evaluation methods from the user perspective, with a specific focus on the public sector. Accordingly, this scoping review aimed to synthesize the scientific literature on evaluating IS from a user perspective, examining the characteristics, development stages, and measurement properties of these evaluation frameworks.

2 Methods

This scoping review was conducted in accordance with Joanna Briggs Institute guidelines and the PRISMA Extension for Scoping Reviews (PRISMA-ScR). The study protocol was registered on the Open Science Framework (https://doi.org/10.17605/OSF.IO/H89T2). In August 2024, a comprehensive search was conducted across several databases, specifically: the ACM Digital Library, Embase, Engineering Village, ERIC, IEEE Xplore, Scientific Health Information from Latin America and the Caribbean countries (LILACS), Medical Literature Analysis and Retrieval System Online (MEDLINE) via PubMed, Scopus, and Web of Science (Table 1). The search strategy focused on three main concepts: “information systems,” “information technology,” and “evaluation framework.” Initially, the strategy was implemented in MEDLINE (via PubMed); the search terms were subsequently adapted to meet the specific requirements of each database. Additionally, two separate searches were conducted on Google Scholar, one in English and the other in Portuguese, screening only the first 100 articles from each search. No restrictions or filters were applied during the database searches. The retrieved articles were then exported to Rayyan® (Qatar Computing Research Institute, Qatar).

Afterward, Reviewer 1 excluded duplicate records by utilizing the platform’s automatic deduplication feature, followed by manual verification. Subsequently, Reviewer 2 was added to the Rayyan project with the blinding feature enabled. In the first eligibility phase, titles and abstracts were screened using the inclusion criteria (Table 2). Initially, the two reviewers independently excluded 10 articles, followed by a consensus meeting. Thereafter, all titles and abstracts were screened independently, and disagreements were resolved by consensus. In the second eligibility phase, full texts were evaluated using a similar approach of independent assessment and consensus. Full texts that were not readily available were requested from the corresponding authors via three attempts over a three-week period. The reference lists of the included full texts were also reviewed.

There were no restrictions regarding the language, geographic location, or publication date of the studies. However, gray literature (e.g., letters, book chapters, non-peer-reviewed conference papers, opinion pieces, articles describing techniques, panels, preprints, and clinical or technical guides) was excluded. Full texts that remained unavailable after three attempts to contact the corresponding authors over a three-week

Table 1. Search strategy in the databases (search on August 31, 2024).
DatabaseSearch strategyRecords
ACM Digital Library[[Abstract: "information systems"] OR [Abstract: "information system"] OR [Abstract: technology] OR [Abstract: technologies]] AND [[Abstract: "evaluation framework"] OR [Abstract: "evaluation frameworks"] OR [Abstract: "structured evaluation"] OR [Abstract: "framework development"] OR [Abstract: "software validation"] OR [Abstract: " software validity "] OR [Abstract: "software verification"]]693
Embase(('information system'/de OR 'information systems':ti,ab,kw OR 'information system*':ti,ab,kw OR 'information technology'/de OR 'technology'/de OR technolog*:ti,ab,kw) AND 'evaluation framework':ti,ab,kw OR 'evaluation framework*':ti,ab,kw OR 'structured evaluation':ti,ab,kw OR 'framework development':ti,ab,kw OR 'software validat*':ti,ab,kw OR 'software validation':ti,ab,kw OR 'software verificat*':ti,ab,kw) AND [embase]/lim2732
Engineering Village((({Information Systems} WN CV) OR ({information management} WN CV) OR ({management information systems} WN CV) OR ({Information System*} WN KY) OR ({Information Technolog*} WN KY) OR ({35} WN ICS) OR ({Technolog*} WN KY))) AND ( (({Evaluation framework*} WN KY) OR ({structured evaluat*} WN KY) OR ({Framework Development} WN KY) OR ({software validat*} WN KY) OR ({software verificat*} WN KY) OR ({Software quality} WN CV) OR ({Computer software selection and evaluation} WN CV) OR ({03.120.01} WN ICS) OR ({03.120.99} WN ICS)))4888
ERIC(descriptorx:"Information Systems" OR descriptorx:"Information Technology" OR descriptorx:"Technology" OR descriptorx:"Educational Technology" OR title:"information system" OR title:"technology" OR abstract:"information system" OR abstract:"technology") AND (descriptorx:"Computer Software Evaluation" OR descriptorx:"Computer Software Reviews" OR title:"Evaluation framework" OR title:"structured evaluation" OR title:"Framework Development" OR title:"software validation" OR title:"software verification" OR abstract:"Evaluation framework" OR abstract:"structured evaluation" OR abstract:"Framework Development" OR abstract:"software validation" OR abstract:"software verification")1436
IEEE Xplore("All Metadata":"Information System*" OR "All Metadata":"Information Technolog*" OR "All Metadata":Technolog*) AND ("All Metadata":"Evaluation framework" OR "All Metadata":"structured evaluat*" OR "All Metadata":"Framework Development" OR "All Metadata":"software validat*" OR "All Metadata":"software verificat*")2157
LILACS(tw:("Information Systems" OR "information system" OR "information systems" OR Technology OR technologies OR "Sistemas de Informação" OR "Sistema de Informação" OR "ICT" OR "TIC" OR "Tecnologia" OR "Tecnologias" OR "Sistema Tecnológico" OR "Sistemas Tecnológicos" OR "Sistemas de Información" OR "sistema de información" OR Tecnología OR Tecnologías OR "Systèmes de recherche d'information" OR "Systèmes d'information" OR Technologie OR Technologies)) AND (tw:("Evaluation framework" OR "Evaluation frameworks" OR "structured evaluation" OR "Framework Development" OR "Modelo de avaliação" OR "Estrutura de avaliação" OR "Avaliação Estruturada" OR "Marco de evaluación" OR "Marco de valoración" OR "Marcos de evaluación" OR "Marcos de valoración" OR "evaluación estructurada" OR "valoración estructurada" OR "Desarrollo del marco" OR "Cadre d'évaluation" OR "Cadres d'évaluation" OR "Évaluation structurée" OR "Élaboration de cadres" OR "software validation" OR "software validations" OR "software verification" OR "software verifications" OR "Validação de Programas de Computador" OR "Validação de Software" OR "Verificação de Programa de Computador" OR "Verificação de Software" OR "Validación de Programas de Computación" OR "Verificación de Programas de Computación" OR "Validation de logiciel" OR "Vérification de logiciel"))92
MEDLINE/PubMed("Information Systems"[MeSH Terms] OR "information system*"[Title/Abstract] OR "Information Technology"[MeSH Terms] OR "Technology"[mh:noexp] OR "technolog*"[Title/Abstract] OR “Digital Technology”[MeSH Terms] OR Educational Technology[MeSH Terms]) AND ("Evaluation framework*"[Title/Abstract] OR "structured evaluat*"[Title/Abstract] OR "Framework Development"[Title/Abstract] OR "software validat*"[Title/Abstract] OR "software validation"[MeSH Terms] OR “software verificat*"[Title/Abstract])975
ScopusTITLE-ABS-KEY ( "Evaluation framework*" OR "structured evaluat*" OR "Framework* Development*" OR "software validat*" OR "software verificat*" ) AND TITLE-ABS-KEY ( "information system*" OR technolog* )3755
Web of Science(TS="Evaluation framework*" OR TS="structured evaluat*" OR TS="Framework* Development*" OR TS="software validat*" OR TS="software verificat*") AND (TS="information system*" OR TS=technolog*)2004
Google Scholar("Information Systems" OR "Information Technology") AND ("Evaluation framework" OR "software validation")100
Google Scholar("Sistemas de Informação" OR "Sistema Tecnológico") AND ( "Modelo de avaliação" OR "Estrutura de avaliação" OR "Validação de Software")100
Source: Elaborated by the authors (2024).

period were also excluded.

Following the selection process, the included articles were divided between two reviewers for data extraction. Four reviewers collectively created the extraction spreadsheet. A pilot extraction was conducted to calibrate the reviewers before the final extraction. Following the extraction process, the two reviewers cross-checked each other’s extracted data. To conclude, a consensus meeting was held.

The extracted characteristics included the first author, year of publication, country of study, study design, name and/or type of IS, name and/or type of evaluation framework, development stage of the evaluation framework, and available psychometric properties. Lastly, the first and second reviewers analyzed the extracted data, followed by verification by all authors. The results are summarized descriptively in the tables and figures provided herein.

Table 2. Eligibility criteria for inclusion in the scoping review.
DomainInclusionExclusion
ProblemCharacteristics, construction stages and/or measurement propertiesNo data on characteristics, construction stages or measurement properties
ConceptEvaluation frameworks1 from the user’s perspectiveMethods not considering the user’s perspective; Methods only addressing the planning and implementation of IS; Studies not presenting the structure of the evaluation framework; Studies not presenting sufficient data on the types of evaluation frameworks
ContextEconomy, agriculture, health, education, environment or infrastructure ISStudies not including IS; Studies only providing data on other technologies
Study typeObservational studies2, clinical trials and systematic reviewsStudies not proposing/presenting an evaluation framework/method; Non-systematic reviews
IS: information systems.
1 Evaluation models, theoretical evaluation frameworks, instruments, questionnaires or any type of tool.
2 Descriptive, cross-sectional and longitudinal.
Source: Elaborated by the authors (2024).

3 Results

Figure 1 presents a summary of the search and selection process. Initially, the searches yielded 18,837 articles, of which 6,011 were excluded as duplicates. Upon reviewing the titles and abstracts, we eliminated an additional 12,780 articles. In the subsequent stage, we assessed 46 articles for potential relevance. However, 34 did not satisfy all inclusion criteria (Table 3). Consequently, 11 studies were included in the review [8-18].

The studies included in this analysis were conducted across various countries, with the United States contributing the most (n = 3); the remaining studies were from Brazil (n = 1) [8], Canada (n = 1) [13], Cyprus (n = 1) [17], India (n = 1) [11], Iran (n = 1) [14], New Zealand (n = 1) [16], Turkey (n = 1) [10], and the United Kingdom (n = 1) [18]. All studies were published in English and from 1999 onward, although most were published after 2006. The primary characteristics of these studies are provided in Table 4.

The studies introduced a range of evaluation frameworks [8-18]. Among the dimensions described in the papers, those assessing quality were the most common, followed by those evaluating user expectations and satisfaction (Figure 2). As for the main characteristics of the frameworks (e.g., methods for application or scoring, etc.) in the sample (Table 5), five reported validating their evaluation frameworks [9,10,15,17,18], four only provided a description of the framework [8,13,14,16], and two developed and applied a framework without validation [11,12]. All studies provided an evaluation framework for IS in healthcare [8-10,12-18], except for one [11], which applied its framework in the healthcare sector but also extended its use to IS in various other fields, such as manufacturing, banking and finance, consulting, commerce,

Figure 1. Flowchart of the article selection process.
Flowchart of the article selection process 2026-11-16T21:39:11.541665 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/
Source: Elaborated by the authors (2024), based on PRISMA 2020.

travel agencies, and education.

The Clinical Information Systems Success Model (CISSM) assesses the effectiveness of clinical information systems across four principal dimensions: system performance, information quality, social influence, and facilitating conditions [9]. The Human, Organization, and Technology-fit (HOT-fit) evaluation framework for HIS integrates human, organizational, and technological factors to assess IS success, underscoring the critical need for alignment among these components [18]. The Development of an Evaluation Framework for Health Information Systems (DIPSA) Framework [17] offers a structured approach to HIS assessment, focusing on performance, organizational impact, and perceived benefits from various stakeholders’ perspectives. Meanwhile, the Electronic Health Record Nurse Satisfaction (EHRNS) framework [15], a questionnaire-based tool, measures nurses’ satisfaction with EHR systems. The Expectation Questionnaire [10] evaluates end users’ expectations of IS to identify discrepancies between expected and actual outcomes.

Table 3. Reasons for exclusion of the studies assessed for eligibility.
Reason for exclusionnExcluded studies (first author, year)
1. Non-systematic reviews or systematic reviews without an evaluation method; letters; book chapters; non-peer-reviewed conference papers; opinion articles; articles describing techniques; panels; preprints and clinical/technical guides5Xydopoulos (2013); Christopoulou (2018); Borim (2015); Palvia (2001); Lee (2010)
2. Studies that do not include information systems or that only provide data on another type of technology11Hub (2008); Wang (2014); Lima (2020); Sabau (2009); Cho (2010); Palacio (2010); Shaw (2001); Cresswell (2020); Sockolow (2012); Marthandan (2010); Usener (2012)
3. Evaluation methods that do not consider the user's perspective3Beuscart-Zéphir (1997); Lau (2009); Smith (2011)
4. Studies that do not present the structure of the evaluation framework5Khairata (2019); Stylianides (2019); Heidebrecht (2014); Grant (2002); Baroudi (1988)
5. Studies that do not present the structure of the evaluation framework in the areas of economy, agriculture, health, education, environment and infrastructure7Jiang (1999); Abidin (2019); Andersson (1997); Al-Tunisi (1995); Russo (2018); Helwig (2020); Goodhue (1998)
6. Studies that do not provide enough data about the types of evaluation frameworks for information systems1Kushniruk (2003)
7. Studies that address only the evaluation of the planning and implementation of information systems, without post-implementation evaluation2Boland (2014); Goldberg (2013)
Total34
Source: Elaborated by the authors (2024).
Table 4. Main characteristics of the included studies (n = 11).
StudyParticipant characteristicsValidityReliability
Garcia-Smith (2013) [9] United States. Design: Literature review and methodological study. IS: Clinical Information System. Framework: Clinical Information Systems Success Model (CISSM). Stage: Validated instrument.The population consisted of 234 nurses. The majority were female (91.0%), with an average age of 28 years and 10.4 years of professional experience.Construct validation; Loading factors: System Performance: 0.726-0.839; Information Quality: 0.835-0.882; Social Influence: 0.679-0.841; Facilitating Conditions: 0.713-0.796Internal Consistency; Cronbach's Alpha: System Performance: 0.80; Information Quality: 0.89; Social Influence: 0.85; Facilitating Conditions: 0.89
Gürsel (2014) [10] Türkiye. Design: Methodological study. IS: Hospital Information System. Framework: Expectation Questionnaire. Stage: Validated instrument.There were 504 participants from Hospital A and 96 participants from Hospital B. Among them, there were professionals from different areas of the hospital (administrative, health and others). In both hospitals, the majority had higher education and good experience in information systems.NRInternal Consistency; Cronbach's Alpha: Importance weights: 0.881; Expectation ratings: 0.970
Kanungo (1999) [11] India. Design: Literature review and development of an evaluation framework. IS: Manufacturing, banking and finance, consultancy, trade, travel agencies, education, and hospitals systems. Framework: IS effectiveness model. Stage: Instrument constructed and applied, but not validated.NANANA
Kukhareva (2022) [12] United States. Design: Literature review and development of an evaluation framework. IS: Epic® EHR. Framework: Evaluation in Life Cycle of Information Technology (ELICIT) framework. Stage: Instrument constructed and applied, but not validated.NANANA
Lau (2007) [13] Canada. Design: Systematic review and development of an evaluation framework. IS: Health Information System. Framework: Infoway Benefits Evaluation Framework. Stage: Description of an evaluation framework.NANANA
Morais e Costa (2014) [8] Brazil. Design: Literature review and development of an evaluation framework. IS: Information systems of the Unified Health System. Framework: NR. Stage: Description of an evaluation framework.NANANA
Norouzi (2024) [14] Iran. Design: Literature review and development of an evaluation framework. IS: Health information systems. Framework: NR. Stage: Description of an evaluation framework.NANANA
Sockolow (2011) [15] United States. Design: Literature review and methodological study. IS: Electronic Health Records (EHR). Framework: Electronic Health Record Nurse Satisfaction (EHRNS). Stage: Validated instrument.The instrument was assessed for reliability and validity in five phases. In the pilot test, participants were nurses studying for a postgraduate degree with a mean age of 26 years, 5 years of professional experience and 2 years of experience with an EHR (n = 30). In the other four phases, participants (n = 90) had a mean age of 49 years and 51% of respondents were nurses or nurse practitioners at the EHR site, compared with 60% at the paper-based site. Other respondents included therapists, social workers and physicians. In both sites, respondents had approximately 23 years of healthcare experience, almost half had experience with an EHR and, of those with EHR experience, the mean EHR experience was 4 years.Content validity: Items and instructions clear; minor issues with ceiling effect on "availability."; Construct validity supported within and across dimensions. Three item pairs showed significant correlations in both surveys (e.g., usability–overall satisfaction; team communication–patient safety; team communication–patient outcome), with six pairs significant in only one survey. Interviews corroborated four correlated items (support, team communication, overall satisfaction, clinician involvement) and five additional items (available, problems, usability, complete/correct, efficiency). EHR functionality frequently highlighted by cliniciansInternal consistency for EHR: Kendall's tau: 0.35–0.57
Stockdale (2006) [16] New Zealand. Design: Literature review and development of an evaluation framework. IS: NR. Framework: Content, context and process (CCP). Stage: Description of evaluation framework.NANANA
Stylianides (2018) [17] Cyprus. Design: Cross-sectional and methodological study. IS: Health Information Systems. Framework: DIPSA evaluation framework. Stage: Validated instrument.The population consisted of 234 nurses. The majority were female (91.0%), with an average age of 28 years and 10.4 years of professional experience.NRInternal Consistency; Cronbach's Alpha: Technology System quality: 0.913; Safety: 0.855; Human Factor Collaboration: 0.939; Satisfaction: 0.916; Organization Procedures: 0.898
Yusof (2008) [18] United Kingdom. Design: Literature review and development of an evaluation framework. IS: Health Information Systems (HIS). Framework: Human, Organization, and Technology-fit​ (HOT-fit). Stage: Validated instrument.The sample consisted of 15 health professionals associated with primary care and hospitals.NRNR
IS: information system. Stage: stage of the framework’s construction. NA: not applicable. NR: not reported.
Source: Elaborated by the authors (2024).

The IS Effectiveness Model [11] is a more generic framework for evaluating IS effectiveness by analyzing technical quality, user satisfaction, and the impact on organizational goals. This approach incorporates both internal evaluations (by CEOs, IS managers, and users) and external assessments (by IT experts), culminating in a composite score derived from a formula that integrates these perspectives. The Content, Context, and Process evaluation framework focuses on health interventions [16], examining intervention content, implementation context, and the processes executed to achieve outcomes. Despite its detailed theoretical basis, this framework has not been applied or validated in practice. Likewise, the Infoway Benefits Evaluation Framework targets HIS [13], emphasizing accessibility, quality of care, and operational efficiency, although it also remains unvalidated. The Evaluation in Life Cycle of Information Technology [12], designed to assess IS throughout their lifecycle with a particular focus on the EPIC® EHR system in the United States, aims to evaluate systems from planning through implementation, emphasizing the fulfillment of organizational needs. Moreover, two studies introduced frameworks without designated titles. Morais and Costa [8] presented a framework for evaluating the IS of the Brazilian Unified Health System, identifying critical indicators of product quality. In Iran, Norouzi et al. [14] developed a framework based on literature from 2000 to 2018, proposing indicators related to system performance, organizational aspects, system design, and technology.

What is more, among the surveyed studies (Table 4), only five reported validating their evaluation frameworks [9,10,15,17,18]. However, the study utilizing the HOT-fit framework did not disclose its validation results [18]. While all five studies provided details on participant sample size and profession [9,10,15,17,18], only three included information on participants’ sex, age, and professional experience [9,15,17].

The CISSM framework underwent validation with 234 nurses across four hospitals [9]. Construct validity was assessed through factor analysis, which revealed that items loaded onto single factors for each dimension. System Performance explained 63% of the variance, thus supporting Hypothesis 1, while Information Quality, which considers usefulness, completeness, format, and accuracy, accounted for 75% of the variance, validating Hypothesis 2. Social Influence, characterized by social factors, services, and support, constituted 55% of the variance, confirming Hypothesis 3. Facilitating Conditions, encompassing behavioral control and work processes, represented 56% of the variance, supporting Hypothesis 4. During the integration stage, Information Quality and Social Influence emerged as the primary predictors of System

Figure 2. Frequency of dimensions in the evaluation frameworks.
Frequency of dimensions in evaluation frameworks 2026-11-16T21:39:11.643984 image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/
Source: Elaborated by the authors (2024).

Use Dependency, explaining 25% of its variance. Nurse Satisfaction was influenced by Information Quality, System Performance, and Facilitating Conditions, which collectively accounted for 70% of the variance, with Information Quality acting as the largest contributing factor. At the outcome stage, Nurse Satisfaction predicted 54% of the variance in net benefits, while System Use Dependency accounted for 14%. Greater system integration was associated with increased dependency, satisfaction, and perceived benefits across the hospitals. Reliability was assessed using Cronbach’s alpha, yielding values of 0.80 (system performance), 0.89 (information quality), 0.85 (social influence), and 0.89 (facilitating conditions).

The DIPSA Framework [17] was validated in a study involving 309 healthcare professionals across two public hospitals in Cyprus. Construct validity was assessed through factor analysis, simplifying the original 42 questions across seven categories into 27 questions across five categories. Although the initial factor analysis revealed 10 potential categories, only five met the inclusion criteria, which required each category to retain at least five questions with a factor loading of ≥0.41. These final categories included satisfaction, system quality, collaboration, procedures, and safety. Internal consistency was evaluated using Cronbach’s alpha, yielding the following results: system quality (0.913) and safety (0.855) under the technology factor; collaboration (0.939) and satisfaction (0.916) under the human factor, and procedures (0.898) under the organization factor.

Table 5. Main characteristics of the evaluation frameworks.
Study, evaluation framework and dimensionsItemsOperationalization: total number of items, items per dimension and scoring
Garcia-Smith (2013) [9] Clinical Information Systems Success Model (CISSM). Dimensions: System performance; Information quality; Social influence; Facilitating conditions; CIS use dependency; Nurse satisfaction; Net benefit.A. System Performance: 1. Perceived Ease of Use; 2. Accessibility; 3. Reliability; B. Information Quality; 4. Perceived Usefulness; 5. Content Completeness; 6. Format; 7. Accuracy; C. Social Influence; 8. Social Factors; 9. Service Support; D. Facilitating Conditions; 10. Perceived Behavioral Control; 11. Work Processes; E. CIS Use Dependency; F. Nurse Satisfaction; G. Net BenefitFour dimensions. 26 items and 6-point Likert response; scale anchored by “very strongly disagree and “very strongly agree.”
Gürsel (2014) [10] Expectation Questionnaire. Dimensions: Usage expectations; System and data expectations; Improvement expectations; Managerial expectations.A. Usage expectations; 1. Ease-of-use; 2. Need for training; 3. Help manuals; 4. Speed; 5. User; B. System and data expectations; 6. Consistency; 7. Privacy; 8. Security; 9. Availability; 10. Interoperability; C. Improvement expectations; 11. Improve service quality; 12. Decreasing workload; 13. Bringing positive change; 14. Research facilities support; D. Managerial expectations; 15. Reporting facilities; 16. Decision support; 17. Function SufficiencyUsers are asked to express their expectation rating using the; 5-point Likert scale (strongly agree, moderately agree, not sure, moderately disagree, and strongly disagree); and importance weights using the 5-point Likert scale (very important, important, averagely important, not so; important, and not important), which are our linguistic variables in fuzzy logic operations.; Importance Weights: Each level of importance has an associated fuzzy number: Very important (W₁): Represented by the fuzzy number (0.75, 1, 1).; Important (W₂): Represented by (0.5, 0.75, 1).; Averagely important (W₃): Represented by (0.25, 0.5, 0.75).; Not so important (W₄): Represented by (0, 0.25, 0.5).; Not important (W₅): Represented by (0, 0, 0.25).; Expectation Ratings: The expectation ratings also have corresponding fuzzy numbers: Strongly agree (R₁): Represented by (0.5, 1, 1).; Moderately agree (R₂): Represented by (0, 0.5, 1).; Not sure (R₃): Represented by (-0.5, 0, 0.5).; Moderately disagree (R₄): Represented by (-1, -0.5, 0).
Kanungo (1999) [11] IS effectiveness model. Dimensions: Effectiveness.1. Predicting future trends; 2. Improving decision making; 3. Avoiding problem areas; 4. Increased user satisfaction; 5. Improving systems integration; 6. Improving resource allocation; 7. Improving speed of operations; 8. Facilitating information retrieval; 9. Improving control over systems; 10. Minimizing errors/mistakes in functional areas; 11. Improving systems standardization; 12. Reducing response time to changes in requirements; 13. Improving quality of product/service; 14. Facilitating communications with customers/suppliersRespondents at each level in each organization; were asked to score each factor in Table 4 on a; 10-point scale (10 being most important and 0; being not relevant). In addition, the research; team also organized a one-day workshop of IT; managers and consultants who provided an; independent assessment of the different factors; based on his or her observations. Thus, each; organization has three internal (CEO, IS manager and user) and one external; evaluations. The final evaluation was obtained; by using the following formula: Score final = (score internal + score external)/2 "where score internal = score CEO + Score IS manager + Score User)/3"
Kukhareva (2022) [12] Evaluation in Life Cycle of Information Technology (ELICIT) framework. Dimensions: Society, user, and IT.Measure Levels; Society; Implementation outcomes: System reach; System adoption; Process outcomes: Guideline compliance; Process efficiency; Health outcomes: Patient health; Equity; Safety; Financial value: Cost; Cost-effectiveness; Return on investment; User; Usability: Ease of use; Ease of understanding; Ease of learning; Workflow: Workflow fragmentation; Number of screens; Cognitive load; System use: Amount/duration; Number of functions used; Number of records assessed; User satisfaction: Perceived usefulness; Enjoyment; Trust in the system; IT; System quality: Performance; Functionality; Security; Information quality: Importance; Relevance; Accuracy; Service quality: Responsiveness; Assurance; Follow-up; Interoperability: Portability; Standards complianceOperation Phase; Technical portability assessment; Definition: Determines the portability of the software across different health systems and EHR platforms.; Methods: Evaluation of interoperability standards (e.g., FHIR), assessment of resources needed for external implementations.; Long-term user satisfaction assessment; Definition: Measures continued user acceptance, usability, and the system's perceived impact on care quality.; Methods: System Usability Scale (SUS) surveys, interviews, and Net Promoter Score (NPS) analysis.; Social outcomes assessment; Definition: Assesses process outcomes, clinical impacts, unintended consequences, health equity, economic outcomes, and the dissemination value of the innovation.; Methods: Clinical trial designs (e.g., before-after, interrupted time series), EHR log analysis, cost-effectiveness analysis, and interviews about unintended outcomes.
Lau (2007) [13] Infoway Benefits Evaluation Framework. Dimensions: System quality; Information quality; Service quality; System use; User satisfaction; Net benefits.A. System quality: 1. Functionality; 2. Performance; 3. Security; B. Information quality: 4. Content; 5. Availability; C. Service quality: 6. Responsiveness; D. System Use: 7. Use behavior/pattern; 8. Self-reported use; 9. Intention to use; E. User satisfaction: 10. Competency; 11. User satisfaction; 12. Ease of use; F. Net benefits: 13. Quality: patient safety; appropriateness/effectiveness; health outcomes.; 14. Access: Ability of patients/providers to access services; patient and caregiver participation.; 15: Productivity: efficiency; care coordination; net cost.NR
Morais e Costa (2014) [8] NR. Dimensions: Product quality.A. Product quality; 1. Performance efficiency; 2. Usability; 3. Security; 4. Maintainability; 5. PortabilityNR
Norouzi (2024) [14] NR. Dimensions: System performance; Organization; System design; Technology.System Performance; Criteria: Efficiency; Effectiveness; Compatibility; Data Security; Quality and Safety; 2. Organization; Criteria: Attitude; Management; Cost; Workflow; 3. System Design; Criteria: Usability; 4. Technology; Criteria: Database Design; Software; Hardware; Communications; System Security; Software DesignIt as a comprehensive hierarchical set of evaluation indicators for assessing health information systems, developed using the Analytical Network Process (ANP) method. This model includes: 4 dimensions,; 16 criteria,; 47 markers,; 180 indicators.
Sockolow (2011) [15] Electronic Health Record Nurse Satisfaction [EHRNS]. Dimensions: Structural quality; Quality of information logistics; Effects on quality of processes; Effects on outcomes and quality of care; Unintended consequences or benefits; Barriers or facilitators to clinicians’ adoption.A. Structural quality; 1. The patient health record is consistently available; 2. The patient health record is subject to frequent system problems*; 3. The patient health record is user-friendly; 4. Sufficient support is available to operate the patient’s health record; 20. Sufficient resources are provided for me to learn to use the patient health record; B. Quality of information logistics; 5. Patient care data being recorded are accurate and valid; 6. Patients have concerns about security or confidentiality*; 13. The patient health record is worth the time and effort required to use it; 14. Overall, I am satisfied with the patient health record; 15. I think patients are satisfied with clinician use of the patient health record; C. Effects on quality of processes; 7. Patient care services are provided in a timely manner; 8. Patient care orders are appropriate; 10. Supports effective communication between most team members about patient care; 16. My department had a role in the introduction of the patient health record at my facility; 17. People who use the patient health record should have had more to say about its design*; D. Effects on outcomes and quality of care; 9. Contributes to the safety of patients; 11. Contributes to patient outcomes; 12. Contributes to patient’s knowledge of their health condition; 21. Part of the increase in costs of healthcare is because of computers*; E. Unintended consequences/benefits; 18. I have firsthand knowledge that problems with the patient health record have interfered with patient care*; F. Barriers/facilitators adoption; 19. My facility is more likely to adopt a computerized patient health record system if the system could exchange patient; information with nursing homes and hospitalsThere are 21 questions divided into six dimensions. The questions are scored using a 5-point Likert scale ranging from 0 = strongly disagree (most dissatisfied) to 5 = strongly agree (most satisfied). Five items (marked with *) were formulated negatively.
Stockdale (2006) [16] Content, context and process (CCP). Dimensions: Content; Context; Process.NANR
Stylianides (2018) [17] DIPSA evaluation framework; D: "Διαδικασίες" (Procedures); I: "Ικανοποίηση" (Satisfaction); P: "Ποιότητα" (System Quality); S: "Συνεργασία" (Collaboration); A: "Ασφάλεια" (Safety). Dimensions: Satisfaction, collaboration,; safety, system quality, procedures.Human Factor: Satisfaction: Measures if users of IHIS are satisfied with the system, considering effort, quality, and performance.; Collaboration: Evaluates whether the system supports collaboration among healthcare professionals.; Technology: System Quality: Assesses aspects such as availability, reliability, access, and the quality of information provided by the system.; Safety: Focuses on how the system enhances patient safety and prevents errors.; Organization: Procedures: Examines the impact of the system on the daily procedures of healthcare professionals.Total Number of Items; The framework includes 42 questions in the initial questionnaire, with 27 questions ultimately grouped into the five main categories (dimensions) based on factor analysis.; 2. Items per Dimension; The five dimensions consist of the following number of items: Satisfaction: 5 items; Collaboration: 5 items; System Quality: 6 items; Safety: 6 items; Procedures: 5 items; 3. Scoring; The questionnaire uses a Likert scale ranging from 1 (lowest value) to 5 (highest value) for most of the items.; Scores for each dimension are calculated based on the responses to the items within that category.
Yusof (2008) [18] Human, Organization, and Technology-fit​ (HOT-fit). Dimensions: Technology; Human; Organization; satisfaction, collaboration,; safety, system quality, procedures.Technology: System Quality; Information Quality; Service Quality; Human: System Use; User Satisfaction; Organization: Organizational Structure; Organizational Environment; Net Benefits: The overall impacts of the system on individual users, organizations, and healthcare deliveryTechnology; System Quality: Data accuracy, data currency, database contents, ease of use, ease of learning, availability, usefulness of system features and functions, flexibility, reliability, technical support, security, efficiency, resource utilization, response time, turnaround time.; Information Quality: Importance, relevance, usefulness, legibility, format, accuracy, conciseness, completeness, reliability, timeliness, data entry methods.; Service Quality: Quick responsiveness, assurance, empathy, follow-up service, technical support.; Human; System Use: Amount/duration (e.g., number of inquiries, amount of connect time, number of functions used, number of records accessed, frequency of access, frequency of report requests, number of reports generated), use by whom? (direct vs. chauffeured use), actual vs. reported use, nature of use (use for intended purpose, appropriate use, type of information used), purpose of use, level of use (general vs. specific), recurring use, report acceptance, percentage used, voluntariness of use, motivation to use, attitude, expectations/belief, knowledge/expertise, acceptance, resistance/reluctance, training.; User Satisfaction: Satisfaction with specific functions, overall satisfaction, perceived usefulness, enjoyment, software satisfaction, decision-making satisfaction.; Organization; Structure: Nature (type, size), culture, planning, strategy, management, clinical process, autonomy, communication, leadership, top management support, medical sponsorship, champion, mediator, teamwork.; Environment: Financing source, government, politics, localization, competition, inter-organizational relationship, population served, external communication.; Net Benefits; Clinical Practice: (Job effects, task performance, productivity, work volume, morale), efficiency, effectiveness (goal achievement, service), decision-making quality (analysis, accuracy, time, confidence, participation), error reduction, communication, clinical outcomes (patient care, morbidity, mortality), cost.
NA: not applicable. NR: not reported.
Source: Elaborated by the authors (2024).

The EHRNS framework [15] was validated in the context of EHR implementation by surveying 37 participants at 11 months post-implementation and 32 participants at 17 months post-implementation. The instrument was deemed clear with relevant items; however, the ‘availability’ item demonstrated a ceiling effect in both samples, indicating a need for revision. Construct validity was partially supported, with significant correlations observed between three item pairs across dimensions (e.g., ‘team communication’ with ‘patient safety’ and ‘patient outcome’), suggesting that the framework would benefit from further validation. Qualitative observations and interviews corroborated the validity of items such as ‘support’ and ‘overall satisfaction.’ Internal consistency, assessed using Kendall’s tau, revealed significant correlations in three of the five related item pairs (e.g., ‘support-resources’ and ‘overall satisfaction-worthwhile’), indicating moderate reliability. The variability in responses reflected diverse user perceptions of satisfaction.

The Expectation Questionnaire [10] was validated by analyzing the importance weights and expectation ratings provided by hospital IS users across two hospitals in Turkey. The validation involved a sample of 600 participants (504 from Hospital A and 96 from Hospital B), with response rates of 76.4% and 96.0%, respectively. The importance weights reflect the relevance users assign to each evaluated variable, measured on a 5-point Likert scale ranging from ‘not important’ to ‘very important.’ Conversely, expectation ratings gauge the extent to which user expectations are met, also evaluated on a 5-point Likert scale from ‘strongly disagree’ to ‘strongly agree.’ To facilitate more nuanced analyses and precise computations of the expectation-meeting index, these data were transformed into fuzzy triangular numbers. The internal consistency of these metrics was assessed using Cronbach’s alpha. For importance weights, the coefficients were 0.871 for Hospital A and 0.942 for Hospital B. For expectation ratings, the scores were 0.966 for Hospital A and 0.959 for Hospital B. The overall Cronbach’s alpha values for these two dimensions were 0.881 and 0.970, respectively.

4 Discussion

This review identified 11 evaluation frameworks for IS from the users’ perspective, mainly within the health domain. The frameworks showed diverse characteristics, underscoring the complexity and multidimensionality of IS. Differences in focus and scope reflect varied methodological approaches, demonstrating adaptability to different contexts. However, their practical utility is frequently limited by a lack of rigorous validation. Among the few studies that reported measurement properties, empirical assessments were often incomplete, thereby reducing both the generalizability of the frameworks and confidence in their findings.

Most studies focused on quality, satisfaction, and user expectations, though they differed in their specific approaches. A common feature was the emphasis on multiple dimensions (e.g., technical quality, user satisfaction, and organizational impact), highlighting the multifaceted nature of IS. Frameworks such as CISSM [9] and HOT-fit [18] stressed the alignment of human, organizational, and technological factors, recognizing that IS effectiveness depends on the continuous interaction among these elements. Other frameworks, including the IS Effectiveness Model [11] and DIPSA [17], assessed system impact on organizational goals and processes to ensure strategic and operational alignment. Notably, DIPSA [17] explicitly builds upon the core variables established in HOT-fit [18].

The evaluated frameworks also differ significantly in scope. DIPSA prioritizes performance and stakeholder benefits, whereas the Infoway Benefits Evaluation Framework [13] focuses on accessibility and quality of care. The Content, Context, and Process framework [16] specifically analyzes health interventions, whereas the IS Effectiveness Model [11] adopts a broader perspective applicable outside the healthcare sector. These variations show that although all the frameworks aim to assess IS, their methods and scopes are tailored to address distinct organizational needs.

Only five studies validated their frameworks [9,10,15,17,18], and one of these failed to report its empirical results. Despite showing promise in terms of reliability and construct validity, the validation efforts in these studies often lacked methodological depth and breadth. Although all used established methods (e.g., factor analysis), the underlying rationale for initial dimension definition was often unclear. Furthermore, factor analysis alone does not ensure robust validation, especially given the requirement for ongoing system adaptation [19].

Content validation was similarly insufficient, typically relying only on expert input without involving end users or accounting for cultural adaptation. This oversight weakens generalizability and undermines confidence in the overarching conclusions, underscoring the need for rigorous empirical validation to ensure accuracy across diverse contexts. The findings also stress the inclusion of core dimensions such as usability, satisfaction, information quality, and security, which collectively reflect both technical efficiency and successful integration into organizational and user environments [20]. While IS adoption is often studied through the lens of specific barriers and facilitators, advancing the theoretical understanding of system adoption, dissemination, and long-term maintenance remains essential. Evaluation models have thus evolved to prioritize key determinants such as perceived usefulness and ease of use [21].

The included studies covered a wide geographic range, and all were published in English. A significant portion of the research was conducted in the United States [9,12,15], with the remaining studies being in Brazil, Canada, Cyprus, Iran, New Zealand, Turkey, and India. This distribution highlights a persistent disparity in the geographic representation of user-centered evaluations for HIS [22].

Despite employing broad search strategies, all included studies ultimately focused on HIS [8-18]. This dominance suggests a gap in the development of user-centered evaluation frameworks within other IS domains. Although one analyzed framework was structurally generic, most addressed specific healthcare contexts, indicating a clear need to extend these evaluative applications to other sectors, such as education and public administration.

Furthermore, despite core dimensions (e.g., interoperability, standards adoption, semantic consistency, and legacy data integration) being widely recognized as critical for the performance and scalability of HIS, these technical aspects are rarely captured by evaluations grounded in the end-user perspective. User-centered assessments tend to focus on attributes directly experienced during daily system interaction, including quality, functionality, usability, and perceived usefulness.

Technical and architectural characteristics (e.g., interoperability or semantic alignment) typically require specialized analyses conducted by system developers, IT managers, or interoperability experts; thus, they fall outside the direct evaluative capacity of most end users. Viewed from this perspective, the predominance of generalized evaluation dimensions identified in this review does not necessarily reflect a limitation of the primary studies, but rather an intrinsic characteristic of user-based evaluation methods.

This scoping review is subject to certain limitations. Extensive methodological variability across studies hindered direct comparability and analytical consistency. Differences in validity assessments and a lack of detailed data (e.g., missing item descriptions, incomplete framework operationalization, and unpublished validation results) further reduced transparency and replicability. Additionally, publication bias may have influenced the synthesized findings. Nonetheless, the review’s strengths include a comprehensive scoping search strategy and the timely relevance of the topic given the accelerating global adoption of health information technologies.

5 Conclusion

This scoping review provides a comprehensive analysis of IS evaluation models, particularly within the healthcare sector. It highlights the prevailing methodological diversity, the limited generalizability of existing tools, and the absence of standardized evaluation dimensions. Although the included studies cover varied approaches and contexts, thereby enhancing our understanding of current evaluation practices, significant challenges remain in result comparability, analytical transparency, and generalizability. These limitations emphasize the urgent need for further research utilizing standardized methodologies, broader demographic populations, and longitudinal study designs. Nevertheless, this review notes important strengths in the literature, including the high clinical relevance of the evaluated topics and the broad range of dimensions addressed, advancing knowledge on the reliability and validity of health evaluation models.

Future research should focus on refining specific evaluation models and would benefit from integrating user-centered evaluation approaches with complementary technical or organizational assessments, such as those related to system integration or interoperability, to provide a more comprehensive understanding of HIS and their downstream implications for healthcare management and service quality.

Lastly, bridging the gap between evaluation frameworks developed in academic research and their practical adoption by the organizations that develop or deploy HIS may contribute to more systematic assessments, improve end-user satisfaction, and enhance overall productivity in real-world clinical settings.

Author contributions. The authors contributed equally to all stages of the article: conception and design of the study; data collection, analysis, and interpretation; drafting and critical revision of the manuscript; and approval of the final version.
Acknowledgments. The authors would like to thank Karyn Munik Lehmkuhl for her support with the search strategies and Atlas Assessoria Linguística for language editing.
Funding. The authors did not declare any source of funding.
Data and materials availability. The authors did not provide a data and materials availability statement.
Conflicts of interest. The authors did not declare any conflicts of interest.
Use of artificial intelligence. The authors did not declare the use of artificial intelligence tools.

References

  1. Pleskach V, Pleskach M, Zelikovska O. Information security management system in distributed information systems. In: 2019 IEEE International Conference on Advanced Trends in Information Theory (ATIT); 2019. p. 300-3. doi: https://doi.org/10.1109/ATIT49449.2019.9030484
  2. Gregersen E. 5 components of information systems. Encyclopaedia Britannica; 2021.
  3. Stoykova S, Shakev N. Artificial intelligence for management information systems: opportunities, challenges, and future directions. Algorithms. 2023;16(8):357. doi: https://doi.org/10.3390/a16080357
  4. Maia MAQ, Barbosa RR, Williams P. Usabilidade e experiência do usuário de sistemas de informação: em busca de limites e relações. Ciênc Inf Rev. 2020;6(3):34-48. doi: https://doi.org/10.28998/cirev.2019v6n3c
  5. BSI. ISO 9241-210:2019. Ergonomics of human-system interaction. Part 210: Human-centred design for interactive systems. London: BSI; 2019.
  6. Cintho LM, Machado RR, Moro CMC. Métodos para avaliação de sistema de informação em saúde. J Health Inform. 2016;8(2). Disponível em: https://jhi.sbis.org.br/index.php/jhi-sbis/article/view/346
  7. Osborne SE. Evaluating health care information systems: methods and applications. J Adv Nurs. 1994;20(1):203-4. doi: https://doi.org/10.1046/j.1365-2648.1994.20010202-6.x
  8. Morais RM, Costa AL. Um modelo para avaliação de sistemas de informação do SUS de abrangência nacional: o processo de seleção e estruturação de indicadores. Rev Adm Pública. 2014;48(3):767-93. doi: https://doi.org/10.1590/0034-76121512
  9. Garcia-Smith D, Effken JA. Development and initial evaluation of the Clinical Information Systems Success Model (CISSM). Int J Med Inform. 2013;82(6):539-52. doi: https://doi.org/10.1016/j.ijmedinf.2013.01.011
  10. Gürsel G, Zayim N, Gülkesen KH, Arifoğlu A, Saka O. A new approach in the evaluation of hospital information systems. Turk J Elec Eng Comp Sci. 2014;22(1):214-22. doi: https://doi.org/10.3906/elk-1110-15
  11. Kanungo S, Duda S, Srinivas Y. A structured model for evaluating information systems effectiveness. Syst Res Behav Sci. 1999;16(6):495-518. doi: https://doi.org/10.1002/(SICI)1099-1743(199911/12)16:6<495::AID-SRES238>3.0.CO;2-R
  12. Kukhareva PV, Weir C, Del Fiol G, Aarons GA, Taft TY, Schlechter CR, et al. Evaluation in Life Cycle of Information Technology (ELICIT) framework: supporting the innovation life cycle from business case assessment to summative evaluation. J Biomed Inform. 2022;127:104014. doi: https://doi.org/10.1016/j.jbi.2022.104014
  13. Lau F, Hagens S, Muttitt S. A proposed benefits evaluation framework for health information systems in Canada. Healthc Q. 2007;10(1):112-6, 118.
  14. Norouzi H, Mehrolhassani MH, Hajesmaeel-Gohari S, Ahmadian L, Ghaemi MM, Mohammadi M, et al. Identification and classification of indicators for evaluating health information systems. Health Policy Technol. 2024;13(2):100833. doi: https://doi.org/10.1016/j.hlpt.2023.100833
  15. Sockolow PS, Weiner JP, Bowles KH, Lehmann HP. A new instrument for measuring clinician satisfaction with electronic health records. Comput Inform Nurs. 2011;29(10):574-85. doi: https://doi.org/10.1097/NCN.0b013e31821a1568
  16. Stockdale R, Standing C. An interpretive approach to evaluating information systems: a content, context, process framework. Eur J Oper Res. 2006;173(3):1090-102. doi: https://doi.org/10.1016/j.ejor.2005.07.006
  17. Stylianides A, Mantas J, Roupa Z, Yamasaki EN. Development of an evaluation framework for health information systems (DIPSA). Acta Inform Med. 2018;26(4):230-4. doi: https://doi.org/10.5455/aim.2018.26.230-234
  18. Yusof MM, Kuljis J, Papazafeiropoulou A, Stergioulas LK. An evaluation framework for health information systems: human, organization and technology-fit factors (HOT-fit). Int J Med Inform. 2008;77(6):386-98. doi: https://doi.org/10.1016/j.ijmedinf.2007.08.011
  19. Reichenheim M, Bastos JL. What, what for and how? Developing measurement instruments in epidemiology. Rev Saude Publica. 2021;55:40. doi: https://doi.org/10.11606/s1518-8787.2021055002813
  20. Barzekar H, Ebrahimzadeh F, Luo J, Karami M, Robati Z, Goodarzi P. Adoption of hospital information system among nurses: a technology acceptance model approach. Acta Inform Med. 2019;27(5):305-10. doi: https://doi.org/10.5455/aim.2019.27.305-310
  21. Ammenwerth E. Technology acceptance models in health informatics: TAM and UTAUT. Stud Health Technol Inform. 2019;263:64-71. doi: https://doi.org/10.3233/SHTI190111
  22. Rana RH, Alam K, Gow J. Health outcome and expenditure in low-income countries: does increasing diffusion of information and communication technology matter? Inf Technol Dev. 2020;26(3):506-24. doi: https://doi.org/10.1080/02681102.2019.1678455

Editorial notes

The Journal of Health Informatics is a scientific journal maintained by the Brazilian Society of Health Informatics (Sociedade Brasileira de Informática em Saúde), a non-profit scientific association dedicated to the development of Health Informatics and Digital Health in Brazil. The editorial process is conducted with scientific independence, free from institutional, commercial, or sponsor interference, and follows international best practices in publication ethics.

This article was double-blind peer reviewed by specialists in the field. The editorial team used artificial intelligence tools, under human supervision, in the initial screening, in checking the text, references and numbers and, when needed, in redrawing figures, charts and tables, which the authors approved in the proof. The tools and safeguards are described in the journal's Policy on the use of artificial intelligence. Editorial decisions were made by people.