Usability evaluation of the Food and Nutrition Monitoring System

VOLTAR AO SUMÁRIO

Avaliação de usabilidade do Sistema de Vigilância Alimentar e Nutricional

Evaluación de usabilidad del Sistema de Vigilancia Alimentaria y Nutricional

DOI: 10.59681/2175-4411.v17.2025.1398

Magda Maria Farias Martins
University of Fortaleza, Postgraduate Program in Public Health, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-5755-0850

Brena Barreto Barbosa
University of Fortaleza, Postgraduate Program in Public Health, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-1536-614X

Yuri Nekan Soares Fontes
University of Fortaleza, Center for Technological Sciences, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-6501-7810

Davi Lopes Teixeira
University of Fortaleza, Center for Technological Sciences, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-1246-502X

José Fernando Rodrigues Ferreira Neto
University of Fortaleza, Center for Technological Sciences, Fortaleza, CE, Brazil.
https://orcid.org/0000-0003-0930-8207

Matheus Leite Pirani Mafra
University of Fortaleza, Center for Technological Sciences, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-5099-2111

José Eurico Vasconcelos Filho
University of Fortaleza, Center for Technological Sciences, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-6881-0814

Claudia Machado Coelho Souza de Vasconcelos
Ceará State University, Postgraduate Program in Nutrition and Health, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-3395-6143

Lia Silveira Adriano
University of Fortaleza, Health Sciences Center, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-7329-6178

Antônio Augusto Ferreira Carioca
University of Fortaleza, Postgraduate Program in Public Health, Fortaleza, CE, Brazil.
https://orcid.org/0000-0002-1194-562X

Autor Correspondente: Antônio Augusto Ferreira Carioca

Artigo recebido: 24/10/2024 | Aprovado: 22/06/2025

ABSTRACT

Objective: To evaluate the quality of interactions between users and the SISVAN (Health Information System for Food and Nutrition Monitoring) Web platform by identifying possible usability problems during system use. Method: This mixed-methods study employed heuristic and usability evaluations to assess the SISVAN Web platform. Researchers experienced in web applications conducted the heuristic evaluation based on Nielsen's heuristics for each system interface. A task-based roadmap was developed for the usability assessment, highlighting the main features and involving both new and experienced users. Results: Half of the problems identified based on the heuristics were classified as unimportant or cosmetic. Frequent technical errors were observed in all usability tests. Some design flaws made the system confusing, particularly during the registration process. These issues persisted regardless of user experience. Conclusions: The system's usability issues affect both new and experienced users, highlighting the need for regular software evaluation, updates, and frequent user training.

Descriptors: Information Systems; Usability Testing; Nielsen's Heuristic Evaluation.

RESUMO

Objetivo: Avaliar a qualidade das interações entre os usuários e o SISVAN (Sistema de Informação em Saúde para Vigilância Alimentar e Nutricional) Web, identificando possíveis problemas de usabilidade. Método: Esta pesquisa qualitativa utilizou avaliações heurísticas e de usabilidade. Pesquisadores experientes em aplicações web realizaram a avaliação heurística com base nas heurísticas de Nielsen para cada interface do sistema. Um roteiro de atividades foi desenvolvido para a avaliação de usabilidade, destacando as principais funcionalidades e envolvendo usuários novos e experientes. Resultados: Metade dos problemas identificados nas heurísticas foram classificados como sem importância ou questões cosméticas. Em todos os testes de usabilidade, foram notados erros técnicos frequentes e alguns erros de design deixaram o sistema confuso, especialmente no cadastro. Esses problemas persistiram independentemente da experiência do usuário. Conclusões: Os problemas de usabilidade do sistema afetam usuários novos e experientes, destacando a necessidade de avaliação regular do software, atualizações e treinamento dos usuários.

Descritores: Sistemas de Informação; Teste de Usabilidade; Avaliação Heurística de Nielsen.

RESUMEN

Objetivo: Evaluar la calidad de las interacciones entre los usuarios y el SISVAN (Sistema de Información en Salud para la Vigilancia Alimentaria y Nutricional) Web, identificando posibles problemas de usabilidad. Método: Esta investigación cualitativa utilizó evaluaciones heurísticas y de usabilidad. Investigadores en aplicaciones web realizaron la evaluación heurística basada en las heurísticas de Nielsen. Se desarrolló un guion de actividades para la evaluación de usabilidad, destacando las principales funcionalidades e involucrando a usuarios nuevos y experimentados. Resultados: La mitad de los problemas identificados en las heurísticas fueron clasificados como de poca importancia. En todas las pruebas de usabilidad, se notaron errores técnicos frecuentes y algunos errores de diseño dejaron el sistema confuso. Estos problemas persistieron independientemente de la experiencia del usuario. Conclusiones: Los problemas de usabilidad del sistema afectan tanto a usuarios nuevos como experimentados, destacando la necesidad de evaluación regular del software, actualizaciones y capacitación de los usuarios.

Descriptores: Sistemas de Información; Prueba de Usabilidad; Evaluación Heurística de Nielsen

INTRODUCTION

Health Information Systems (HIS) are used by the Brazilian Unified Health System (SUS) to standardize procedures for collecting, recording, processing, storing, and retrieving data in health services. Their main objective is to produce information necessary for planning, evaluating, and implementing health actions and services from the perspective of individual and collective care ⁽¹⁾.

The Food and Nutrition Monitoring System (SISVAN) is a Health Information System created to continuously monitor and provide information on the nutritional status and food consumption of the Brazilian population. The SISVAN online platform (SISVAN Web) was created in 2008 to consolidate information on the nutritional status and food consumption of SUS users, the world's most extensive public health system, serving more than 200 million people ⁽²⁾. The SISVAN Web stores the records inserted in the Management System of the Bolsa Família Program for Health, in the e-SUS Primary Care (e-SUS APS), and in SISVAN, where specific consolidated reports are available only on SISVAN Web. The data storage and qualification allow for the production of food and nutrition indicators, offering support to professionals and management in the organization of nutritional health care ⁽³⁾.

SISVAN has not been used to its full potential since the deployment of the web platform. This condition is observed by the low recording of information on the nutritional status of adults and older adults ⁽⁴⁾ and food consumption in all age groups in the country ⁽⁵⁾. Some of the main difficulties that the professionals responsible for operating the system pointed out are the lack of technological resources such as a computer, lack of internet access or low-quality connection, and lack of professional qualifications regarding the use of the online system ⁽⁶⁾.

Usability is an important aspect of digital technologies and contributes to digital platforms' quality and ease of use. Evaluating the usability identifies concrete problems in the user's interaction with the system, considering the results of using the current interface, and can lead to the possible construction of new versions that can promote improvements in the use of programs and data evaluation ^(7,8). The lack of professional training to work with websites and online programs can cause an unsatisfactory usability process, resulting in low coverage, lack of integration between existing systems, and a lack of adequate professionals in data collection and analysis ⁽⁹⁾.

Evaluating SISVAN’s usability reinforces the concept that the implementation of Food and Nutrition Monitoring actions demands a reflection on the technical-operational aspects of this system ⁽¹⁰⁾. Proposing strategies to improve the collection and insertion of data into the system presupposes evaluating usability, focusing on analyzing the quality of interactions between users and the system. Thus, this study aimed to evaluate the quality of interactions between users and SISVAN Web through the evaluation of possible usability problems when using the system.

METHODS

This mixed-methods investigation employed the usability test and the heuristic evaluation method proposed by Nielsen ⁽⁸⁾ with professionals who use SISVAN Web and computing specialists. Researchers from the University of Fortaleza conducted the evaluations through the Google Meet video platform with real users who interacted with the system to perform specific tasks in a simulated or real operating context.

The SISVAN Web platform

The first version of SISVAN was computerized and made available by the Ministry of Health in 2004, guided by Decree N° 2246 of October 18, 2004. The main advantages offered were enabling the recording and dissemination of information about the anthropometric assessment and food consumption of the population served in Primary Care, whether this be children, adolescents, adults, older adults, or pregnant women ^(11,12).

In 2017, the 3.0 version of the system was released and made available to streamline the integration of SISVAN with e-SUS Primary Care. Access to the SISVAN system is via the electronic address: https://sisaps.saude.gov.br/sisvan/, where several tabs are organized into the following structure: 1) General aspects; 2) Registration; 3) Monitoring Records; 4) Links; 5) Reports; 6) Integration of SISVAN with other information systems; 7) System Support; and 8) Support materials ⁽¹²⁾.

Food and Nutrition Monitoring recommends the evaluation of nutritional status indicators from anthropometric indices and food consumption markers, and all the professionals from Primary Health Care are responsible for collecting and entering this information ⁽¹¹⁾. The team of primary care professionals must be composed of at least a doctor, a nurse, a nursing assistant and/or technician, and a community health worker ⁽¹³⁾. In 2019, 44,188 health teams were registered in primary care in Brazil ⁽¹⁴⁾. Nutritionists, physiotherapists, dentists, and psychologists, among other specialties, may be included besides the minimum team health professionals ⁽¹³⁾.

Heuristic evaluation

Heuristic evaluation consists of a strategy of analyzing and reviewing user interfaces to discover and correct system problems. This type of evaluation does not presuppose the need for users but for experts who evaluate the interfaces based on heuristics defined in the literature from the field in question ⁽¹⁵⁾.

In SISVAN’s heuristic evaluation, a team of six people (computer science academics and computer scientists) with experience in web applications was included by convenience. Each expert evaluated the system interfaces for each of Nielsen's 10 heuristics, as follows: system status visibility; matching the system and the real world; user control and freedom; consistency and standards; error prevention; recognition rather than recall; flexibility and efficiency of use; aesthetics and minimalist design; assistance to users in recognizing, diagnosing, and correcting errors; and help and documentation. Participants had to inform the degree of severity of any problem, (0 = unimportant; 1 = cosmetic; 2 = simple; 3 = severe; 4 = catastrophic) and leave comments ⁽¹⁵⁾.

Experts evaluated the Access to the System, Individual Grouping, Individual Registration, Nutritional Status Monitoring, and Monitoring Map screens, as these are the screens most used by real users.

Upon completion of the individual assessment, the evaluators were divided into two teams of three for a second assessment to discuss their previous responses with each other until they reached a single-response consensus for the heuristic for each screen. Finally, all the evaluators formed a single team for a third and final evaluation so that they could reach a consensus again. This consensus should be as concise as possible, as it contains the opinions of all the evaluators.

Although the heuristic evaluation was conducted exclusively by computer science professionals, this choice was intentional and methodologically grounded. According to Nielsen's original proposition, usability experts best perform heuristic evaluations and are trained to identify interface design issues based on established principles systematically. Including health professionals in this phase could compromise the consistency of heuristic interpretation, given their limited familiarity with usability standards. However, recognizing the importance of incorporating the perspective of actual or potential system users, we complemented the study with usability testing involving nutrition professionals who had no prior experience with SISVAN Web but were considered potential system users within primary health care. This combination of methods ensured a comprehensive and triangulated system usability analysis.

Usability testing

The usability test ⁽¹⁵⁾ and the heuristic evaluation verify how easy or difficult it is to use the system in question. The ease or difficulty of carrying out a usability test depends on the level of demand required for the results, the generality of the system, and the availability of resources and users ⁽¹⁶⁾. In this evaluation, the focus was on analyzing how new users interact with the system and what their hardest challenges are.

The usability test must be conducted with new users, so six nutritionists with no previous contact with SISVAN Web were chosen by convenience. The test was also conducted with six experienced nurses and nutritionists with at least one year's experience using the SISVAN Web to compare with what would be a "best case".

Initially, a roadmap of activities was developed, divided into tasks, and defined from the main available functionalities, followed by all users participating in the test. When taking the test, the instructor named the tasks individually and noted the time, number of clicks, and user behavior for each task as the user performed them. The behaviors were classified as a priori: performed quickly, took time to achieve, complained, asked for help, and did not achieve.

The users evaluated the System Access screens (task 1: access the system); Grouping of Individuals (task 2.1: search and task 2.2: grouping); Individual Registration (task 3.1: registration; task 3.2: verification and task 3.3: update), Nutritional Status Monitoring (task 4.1: show history and task 4.2: follow-up registration) and Monitoring Map (task 5: download), as these were the screens most used by real users.

At the end, users received a post-test questionnaire with two blocks of statements based on the System Usability Scale (SUS) ⁽¹⁷⁾ (Table 1) to assess each system task and their experience with user-friendly features and challenges. The assessment was performed on a five-point hedonic scale (from “strongly agree” to “strongly disagree”), resulting in a score from 0 to 100 for each user, with 68 being the standard SUS average.

Table 1. Statements answered as a post-test by new and experienced SISVAN Web users.

First affirmations block	Second affirmations block
I found it easy to access the system with my credentials	I easily indicated how I group the individuals
I struggled to access the system with my credentials	I struggled to indicate how to group the individuals
I easily accessed the registration of individuals section	I found it easy to find the nutritional status section
I found it challenging to find the registration of individuals section	It was challenging to find the nutritional status section
I easily verified the existence of the requested user	I easily registered a new monitoring of food consumption for the requested user
I struggled to verify the existence of the requested user	I struggled to register a new monitoring of food consumption for the requested user
I found it easy to register the requested user	I easily found the nutritional status monitoring map section
I found it difficult to register the requested user	I found it challenging to find the nutritional status monitoring map section
I easily found the grouping individuals section	I found it easy to download an Excel file with the requested individuals
I found it challenging to find the grouping individuals section	I downloaded, with difficulty, an Excel file with the requested individuals

Ethical aspects

The procedures for obtaining data, analyzing it, and disseminating the results followed the norms of the National Health Council. All volunteers signed an Informed Consent Term agreeing to participate in the research. The Research Ethics Committee of the University of Fortaleza approved the project under Opinion N° 4.348.452).

RESULTS

Heuristic evaluation

In all the heuristics, at least half of the errors were unimportant, and cosmetic errors were presented. We identified simple errors in 8 out of the 10 categories, serious errors in 6 categories, and catastrophic errors only in one category (Figure 1).

Figure 1. Nielsen’s heuristic analysis of SISVAN Web.

We observed that on the screen “Registration of individuals”, three errors of a cosmetic nature were found: “match between system and the real world”, “user control and freedom”, and “consistency and standards”. “System Status Visibility” and “Help and Documentation” had two simple bugs.

On the “Registration Update” screen, we found two cosmetic errors in system status visibility and recognition instead of recall. On the “Registration” screen, the evaluators identified two serious errors, which were regarding system status visibility and help and documentation. In both errors, the help card did not match the current page, since the title and description referred to the registration update. In addition, a cosmetic error was found in consistency and standards.

On the "Nutritional Status" screen, we found a simple error in system status visibility and three cosmetic errors in user control and freedom, consistency and standards, aesthetics, and minimalist design.

On the "History" screen, we detected only two cosmetic errors in system status visibility and consistency and defaults. Finally, on the "New Monitoring (Nutritional status and Food consumption)" screen showed simple errors about helping users recognize, diagnose, and correct errors, in addition to two cosmetic errors in matching the system with the real world and in help and documentation.

Usability testing

We observed frequent technical errors in all usability tests, which included the following: (1) when clicking on a button, the system redirected the user to the initial screen and, on other occasions, took the user to an error screen. In addition to these technical errors, some design errors left the system confused, mainly concerning registration: (2) when clicking on the registration section, the user was redirected to a page where the focus was the search for a user already registered in the system instead of the registration itself, which confused users and made them complete the fields unnecessarily.

After this step, some users were in doubt about loading the user by employing the National Health Card, as clicking on “Load individual” opened a loading screen that could be closed, meaning that the user did not know that it was still loading and ended up getting lost in the registration, which was compounded by the fact that loading often took a long time. Some users faced another impasse: the checkbox. This confused users by opening a text box below it, making it appear to belong to the previous field. Also, when selecting an option, the list of items continued to be displayed, making users unable to see later items until they clicked on the screen. However, many users were unaware that the system required this. Finally, regarding the multiple selection field, some users did not understand that they could select multiple options.

Table 2 shows that, in a general assessment, the main behaviors and task durations of new and experienced users were similar.

Table 2. Behavior and duration of tasks of new and experienced users.

User Tasks	New Users		Experienced Users
	Most User Behavior	Mean duration	Most User Behavior	Mean Duration
Task 1: Access the system	Accessed the system quickly	01:00	Accessed the system quickly	00:16
Task 2.1: Grouping of individuals – Search	Quickly searched	00:51	Quickly searched	00:29
Task 2.2: Grouping of individuals - Grouping	Informed quickly	00:07	Informed quickly	00:31
Task 3.1: Individual registration - Registration	Took a while to register	04:23	Registered quickly	02:17
Task 3.2: Individual Registration - Verification	Verified quickly	01:05	Verified quickly	00:54
Task 3.3: Individual registration – Update	Found quickly	00:14	Found quickly	00:02
Task 4.1: Nutritional status - Show history	Showed quickly	00:31	Showed quickly	1:25
Task 4.2: Nutritional status - Monitoring registration	Took a while to register	05:15	Registered quickly	02:11
Task 5.1: Monitoring Map – Download	Downloaded quickly	01:19	Downloaded quickly	01:12

After performing the usability test, the volunteers answered the post-test questionnaire. The averages were calculated with these results, considering 68 as the average for the SUS. We obtained an average of 81.78 for the most experienced system users, and this average was 86.45 for new users. We observed that the scores were above average, which leads us to understand that the system is not complex to use. However, both groups' scores were very close, indicating that the usability problems remain similar, regardless of the user's experience with the system.

DISCUSSION

We analyzed the usability of SISVAN Web through analysis with experts, using heuristic principles, and through analysis with real users of the system (primary health care professionals). The main non-conformities in the heuristics were system status visibility, consistency and standards, and help and documentation. In the usability test with real users, we observed system characteristics that hinder the interaction of new and experienced users. However, we observed that half of the errors were unimportant and that there were also cosmetic errors.

Health information system assessments aim to improve the quality of patient care provided by health professionals and improve data collection and analysis performed by managers. According to a systematic review of the literature, most studies that evaluated health information systems adopted usability aspects as a method ⁽¹⁸⁾. Some studies evaluated the usability of Brazilian Information Systems, such as the Immunization Information System ⁽¹⁹⁾, Live Birth Information System ⁽²⁰⁾, and Neonatal Health Information System ⁽²¹⁾. Even though it is the primary method, we observed that usability evaluations are rarely used in health information systems, and we could not find studies evaluating the SISVAN Web.

The main limitations and problems found in other health information systems were related to the perception of the user's location on the route taken within the system itself, standardization of screens, resources made available by the system, in addition to failures and duplicity of commands, which hampers the registration and access to information ⁽¹⁹⁾. Our findings revealed that some errors were more frequent according to users' reports, such as an error when clicking on any system button, causing redirection to the error page, and another was characterized as errors associated with the registration of individuals, in which the page was directed to search individuals.

There may be possible discrepancies between expert and user evaluations. While heuristic errors were minor, users reported challenging system interactions. A systematic review of health information systems highlighted five heuristics with the highest usability problems, including “Flexibility and efficiency of use”, “Consistency and standards”, and “Help and documentation”. These issues protract interactions, causing task delays, dissatisfaction, and frustration, preventing users from fully utilizing system benefits and functionality despite minor heuristic concerns ⁽²²⁾.

Various methods evaluate usability, including questionnaires and direct observation. This study used Heuristic Evaluation and the System Usability Scale (SUS) due to their suitability ⁽²³⁾. Heuristic Evaluation is quick, cost-effective, and standardized but requires a usability expert and may identify irrelevant issues ⁽²⁴⁾. SUS is easy, cost-effective, and freely available but relies on user perception and provides non-specific information on usability issues ⁽²⁵⁾.

Despite the limitations related to the absence of data entry in the system by the experts, which can limit the heuristic evaluation, we can highlight the involvement of real users with and without experience in the system as a strong point.

The effective, efficient, and satisfactory usability of the SISVAN Web enhances decision-making, transforming epidemiological and nutritional indicators into actionable health and nutrition monitoring, thereby improving information quality and population well-being. Usability assessments should focus on user feedback to adapt the interface or develop new versions that meet user expectations and needs.

CONCLUSION

The main non-conformities we identified were system status visibility, consistency and standards, and help and documentation, with half classified as unimportant or cosmetic. Usability tests showed that new and experienced users faced difficulties.

Although experts deemed SISVAN's usability satisfactory, improvements are recommended, including better-defined standards and enhanced system connectivity, as many users encountered connection issues. This study underscores the importance of regular software evaluation, continuous system updates, and frequent user training, regardless of users' experience with the system.

REFERENCES

1. Brasil. Ministério da Saúde. Organização Pan-Americana da Saúde. A experiência brasileira em sistemas de informação em saúde. Brasília: Editora do Ministério da Saúde, 2009 [cited on 2023 April 4]. Available at: https://bvsms.saude.gov.br/bvs/publicacoes/experiencia_brasileira_sistemas_saude_volume1.pdf.

2. Universidade Aberta do Sistema Único de Saúde (UNASUS). Maior sistema público de saúde do mundo, SUS completa 31 anos, 2021 [cited on 1 Feb 2023]. Available at: https://www.unasus.gov.br/noticia/maior-sistema-publico-de-saude-do-mundo-sus-completa-31-anos

3. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Marco de referência da vigilância alimentar e nutricional na atenção básica. Brasília, DF; 2015 2015 [cited on 2 Feb 2023]. Available at: http://189.28.128.100/dab/docs/portaldab/publicacoes/marco_referencia_vigilancia_alimentar.pdf

4. Nascimento FA, Silva SA, Jaime PC, Cobertura da avaliação do estado nutricional no Sistema de Vigilância Alimentar e Nutricional brasileiro: 2008 a 2013. Cad. Saude Publica. 2017; 33(12): e00161516. DOI: 10.1590/0102-311X00161516.

5. Nascimento FA, Silva SA, Jaime PC. Cobertura da avaliação do consumo alimentar no Sistema de Vigilância Alimentar e Nutricional Brasileiro: 2008 a 2013. Rev. Bras. Epidemiol. 2019; 22: e190028. DOI: 10.1590/1980-549720190028.

6. Rolim MD, Lima SML, Barros DCD, Andrade CLT. Avaliação do SISVAN na gestão de ações de alimentação e nutrição em Minas Gerais, Brasil. Ciênc Saúde Colet. 2015; 20(8): 2359-2369. DOI: 10.1590/1413-81232015208.00902015.

7. Associação Brasileira de Normas Técnicas. NBR ISO/IEC 29110: Engenharia De Software Qualidade De Produto. Rio De Janeiro (RJ): ABNT; 2011. [cited on February 2, 2023]. Available at: http://www.abntcatalogo.Com.Br/Norma.Aspx?Id=002815

8. Nielsen J. Nielsen Norman Group. Usability 101: Introduction To Usability; 2012 [cited 2023 April 4]. Available at: https://www.nngroup.com/articles/usability-101-introduction-to-usability/

9. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Indicadores de Vigilância Alimentar e Nutricional: Brasil 2006. Brasília, DF: 2009 [cited on 2023 Apr 5]. Available at: http://189.28.128.100/nutricao/docs/geral/indicadores_vigilancia_alimentar_nutricional.pdf

10. Recine E, Vasconcellos AB. Políticas nacionais e o campo da Alimentação e Nutrição em Saúde Coletiva: cenário atual. Ciênc. Saúde Colet. 2011; 16(1): 73-79, DOI: 10.1590/S1413-81232011000100011.

11. Brasil. Ministério da Saúde. Secretaria de Atenção à Saúde. Departamento de Atenção Básica. Manual operacional para uso do Sistema de Vigilância Alimentar e Nutricional SISVAN – versão 3.0. Brasília, DF: 2017 [cited on 2023 April 7]. Available at: http://sisaps.saude.gov.br/sisvan/public/file/ManualDoSisvan.pdf

12. Mourão E, Gallo CDO, Nascimento FAD, Jaime PC. Tendência temporal da cobertura do Sistema de Vigilância Alimentar e Nutricional entre crianças menores de 5 anos da região norte do Brasil, 2008-2017. Epidemiol. Serv. Saude. 2020; 29(2): e2019377. DOI: 10.5123/s1679-49742020000200026.

13. Brasil. Ministério da Saúde. Portaria n° 2.436, de 21 de setembro de 2017. Aprova a Política Nacional de Atenção Básica, estabelecendo a revisão de diretrizes para a organização da Atenção Básica, no âmbito do Sistema Único de Saúde (SUS), Brasília, DF; 2017 [cited on 2023 Feb 2]. Available from: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2017/prt2436_22_09_2017.html

14. Brasil. Ministério da Saúde. E-Gestor Atenção Básica: espaço para informação e acesso aos sistemas de Atenção Básica. 2019 [cited on 2023 Feb 2]. Available from: https://egestorab.saude.gov.br/paginas/acessoPublico/relatorios/relHistoricoPagamentoEsf.xhtml

15. Rogers Y, Sharp H, Preece J. Design de interação. Porto Alegre: Bookman Editora, 2013. 585 p.

16. Cybis W, Betiol AH, Faust R. Ergonomia e usabilidade: conhecimentos, métodos e aplicações. Novatec editora, 2017.

17. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: Adding an adjective rating scale. J. Usability Studies. 2009; 4(3): 114-123.

18. Cintho LMM, Machado RR, Moro CMC. Métodos para avaliação de sistema de informação em saúde. J. Health Inform. 2016; 8 (2): 41-8.

19. Morato YC, Carvalho BF, Oliveira VC, Dias TMR, Cavalcante RB, Amaral GG et al., Análise do Sistema de Informação em Imunizações do Brasil sob a ótica das heurísticas de usabilidade. Rev. Cubana Inf. Cienc. Salud. 2020; 31: 1-20. DOI: 10.36512/rcics.v31i2.1515.

20. Machado LCB, Meirelles BHS. Avaliação da Usabilidade do Sistema Informações Nascidos Vivos no Estado de Santa Catarina. J. Health Informatics. 2021; 12: 229-34.

21. Padrini-Andrade LL, Balda RCX, Areco KCN, Bandiera-Paiva P, Nunes MDV, Marba STM et al. Avaliação da usabilidade de um sistema de informação em saúde neonatal segundo a percepção do usuário. Rev. Paul. Pediatr. 2018; 37(1): 90-96, DOI: 10.1590/1984-0462/;2019;37;1;00019.

22. Dias CR, Pereira MR, Freire AP. Qualitative review of usability problems in health information systems for radiology. J. Biomed Inform. 2017; 76: 19-33. DOI: 10.1016/j.jbi.2017.10.004

23. Keenan HL, Duke SL, Wharrad HJ, Doody GA, Patel RS. Usability: An introduction to and literature review of usability testing for educational resources in radiation oncology. Tech. Innovat. Pat. Supp. Radiat. Oncol. 2022; 24: 67-72. DOI: 10.1016/j.tipsro.2022.09.001

24. Jaspers MWM. A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int. J. Med. Inf. 2009; 78: 340-353. DOI: 10.1016/j.tipsro.2022.12.002

25. Lewis JR. The System Usability Scale: Past, Present, and Future. Int J Hum-Comput Interact. 2018; 34: 577-590. DOI: 10.1080/10447318.2018.1455307