Comparison of administration methods on OHIP-14 scores

VOLTAR AO SUMÁRIO

Comparação de métodos de administração nas pontuações do OHIP-14

Comparación de métodos de administración en las puntuaciones del OHIP-14

DOI: 10.59681/2175-4411.v17.2025.1153

Bianca Schlesener Dettmer
Department of Restorative Dentistry, Federal University of Santa Maria, Santa Maria, RS, Brazil.
https://orcid.org/0009-0001-1535-4368

Ricardo Kenji Takahama
Department of Stomatology, School of Dentistry, Federal University of Santa Maria, Santa Maria, RS, Brazil.
https://orcid.org/0009-0001-0688-7058

Letícia Donato Comim
Department of Restorative Dentistry, Federal University of Santa Maria, Santa Maria, RS, Brazil.
https://orcid.org/0000-0002-9386-4785

Julio Eduardo do Amaral Zenkner
Department of Stomatology, School of Dentistry, Federal University of Santa Maria, Santa Maria, RS, Brazil.
https://orcid.org/0000-0003-0302-4449

Luana Severo Alves
Department of Restorative Dentistry, Federal University of Santa Maria, Santa Maria, RS, Brazil.
https://orcid.org/0000-0003-0110-7929

Autor Correspondente: Letícia Donato Comim

Artigo recebido: 14/12/2023 | Aprovado: 06/07/2025

ABSTRACT

Objective: Compare two administration methods of Oral Health Impact Profile (OHIP-14), online and telephone. Methods: 109 young adults initially answered OHIP-14 online. After 15 days, participants answered again on telephone. Scores were compared using Wilcoxon test. Cronbach's alpha estimated the reliability and Intraclass Correlation Coefficient evaluated the reproducibility. A simple linear regression was performed to assess systematic bias. Results: Mean (standard deviation [SD]) and median (interquartile range [IQR]) OHIP-14 scores were_ 9.72 (SD 9.71) and 6 (IQR 3; 14) for online and 8.38 (SD 8.43) and 6 (IQR 2; 11) for telephone. Significant differences were found for overall score (p=0.02) and psychological disability domain (p<0.01), with online higher scores. Cronbach's alpha was 0.92, while ICC was 0.76. Linear regression showed that online tended to yield higher scores (p=0.004). Conclusion: Both methods demonstrated high values of reliability and moderate values of reproducibility; however, online systematically resulted in higher scores.

Descriptors: oral health; quality of life; surveys and questionnaires.

RESUMO

Objetivo: Comparar dois métodos de administração do Oral Health Impact Profile (OHIP-14), on-line e por telefone. Métodos: 109 jovens adultos responderam inicialmente ao OHIP-14 on-line. Após 15 dias, responderam novamente por telefone. As pontuações foram comparadas pelo teste de Wilcoxon. O Alfa de Cronbach estimou confiabilidade e o Coeficiente de Correlação Intraclasse (CCI) reprodutibilidade. Uma regressão linear simples avaliou viés sistemático. Resultados: A média (desvio padrão [DP]) e a mediana (intervalo interquartil [IIQ]) dos escores on-line foram 9,72 (DP 9,71) e 6 (IIQ 3; 14) e 8,38 (DP 8,43) e 6 (IIQ 2; 11) com telefone. Foram encontradas diferenças significativas no escore geral (p=0,02) e incapacidade psicológica (p<0,01), com maiores scores online, Alfa de Cronbach de 0,92 e ICC 0,76. Na regressão linear o online tendeu a pontuações mais altas (p=0,004). Conclusão: Ambos métodos demonstraram bons valores de confiabilidade e reprodutibilidade; porém, o online resultou sistematicamente em pontuações mais altas.

Descritores: saúde bucal; qualidade de vida; inquéritos e questionários.

RESUMEN

Objetivo: Comparar dos métodos de administración del Perfil de Impacto en la Salud Oral “(OHIP-14)”, en línea y por teléfono. Métodos: 109 adultos jóvenes respondieron inicialmente OHIP-14 en línea. Pasados 15 días, respondieron, una vez más, por teléfono. Los rangos se compararon por Wilcoxon prueba. Confiabilidad estimada por Alfa de Cronbach y reproductibilidad por Coeficiente de Correlación Intraclase (CCI). Una regresión lineal simple evaluó sesgo sistemático. Resultados: Promedio (desviación estándar [DE]) y mediana (rango intercuartílico [RQ]) de los rangos fueron: 9,72 (DE 9,71) y 6 (RQ 3; 14) en línea y 8,38 (DE 8,43) y 6 (RQ 2; 11) con teléfono. Se encontraron diferencias significativas el rango general (p=0,02) y discapacidad psicológica (p<0,01), con mayores rangos en línea, Alfa de Cronbach 0,92 y ICC 0,76. En regresión lineal, online tendió a tener rangos más altos (p=0,004). Conclusión: Ambos métodos demostraron buena confiabilidad y reproductibilidad; sin embargo, en línea resultó consistentemente en rangos más altas.

Descriptores: salud bucal; calidad de vida; encuestas y cuestionarios.

INTRODUCTION

Oral health-related quality of life (OHRQoL) is a multidimensional construct that includes a subjective evaluation of the individuals’ oral health, functional and emotional well-being, expectations and satisfaction with care, and sense of self ⁽¹⁾. Patients’ oral health status is directly associated with their self-perception of oral health and OHRQoL ^(2).

OHRQoL is assessed through age-specific questionnaires that measure the impact of oral health on the quality of life of individuals^(1,3). The Oral Health Impact Profile (OHIP) is a widely used instrument to measure the OHRQoL of adults, and is usually applied as a face-to-face interview or self-administered personally. However, the complex pandemic scenario that began in 2020 resulting from the spread of the SARS-CoV-2 virus and classified by the World Health Organization (WHO) as “Corona Virus Disease (COVID-19)” ⁽⁴⁾ had a huge impact on health systems and scientific communities around the world ⁽⁵⁾. As a consequence, many surveys began to be administered online through Google Forms questionnaires ^(6,7), which brought a solution for the continuity of remote research, when agreement with validated methods of data collection is proved to exist.

Some previous studies compared different methods of administration of OHRQoL questionnaires ^(8-12), but they were restricted to evaluate self-administration, face-to-face interview, and telephone interview. To the best of our knowledge, there is no previous study comparing methods of online administering the OHIP-14 questionnaire with the methods commonly used, especially among young adults. Therefore, the aim of this study was to compare two methods of administration of the OHIP-14 questionnaire, online administration (OA) via Google Forms and telephone administration (TA). We hypothesized that there was no difference between the methods under investigation.

METHOD

The study protocol was approved by the Research Ethics Committee of the Federal University of Santa Maria (CAAE 43938021.0.0000.5346) and all participants signed an online informed consent form.

This is an analytical observational study. The sample size calculation was performed using the GPower software. Considering an effect size of 0.3, an alpha value of 0.05, and a power of 80%, the minimum sample size required for the present study was 74 participants. A dropout rate of 30% was added, totaling 106 participants to be invited to participate.

In 2018, a population-based cross sectional study was conducted to assess the oral health status of 15-to-19-year-old adolescents attending public and private high schools in Santa Maria, a mid-sized city located in southern Brazil. At that moment, 1,197 individuals were included, resulting in a representative sample of the population. Full details about the methodology used in the epidemiological survey are published elsewhere ⁽¹³⁾. Three years later, in 2021, during the COVID-19 pandemic and under social distancing restrictions, all individuals included in 2018 (n= 1,197) were recontact, by e-mail, Facebook or WhatsApp. The message had an invitation to participate in the present investigation by answering the questionnaires with different application methods, online and by telephone calls. Participation in the 2018 survey served as a criterion for eligibility for invitation. The participants who agreed to participate were included in this survey; no other inclusion/exclusion criteria were applied and the final sample included those who completed the first and second stages. Therefore, the present study included a convenience sample of young adults aged between 18 and 22 years.

Data collection was conducted from April to August 2021, and was performed in two stages: OA via Google Forms and TA. First, the researchers contacted the participants of the epidemiological survey through e-mail and then through social media (Facebook and WhatsApp). For each strategy, three attempts were performed at one-week intervals. The participants received a message containing general information about the research and an invitation to participate. At the end of the message, a link to access the questionnaire was provided.

The online questionnaire was composed of four sections. The first one contained the informed consent form, describing the objectives, justification and research method. The second and third sections contained closed questions about socioeconomic and demographic characteristics, oral hygiene habits, use and access to dental services. The characteristics included were sex (male or female), skin color (dichotomized as a binary variable, non-white or white), age (18, 19, or ≥20 years), level of education (high school, university/technical qualification, or school evasion), family income (≤2 Brazilian Minimum Wages [BMW] or >2 MBW), visit to a dentist in the last six months (yes or no), self-reported gingival bleeding (yes or no), and  tooth brushing frequency (≤twice/day or >twice/day). These variables were collected for sample description purposes.

Finally, in the fourth section, data on the impact of oral conditions on OHRQoL was collected through the Brazilian reduced version of the Oral Health Impact Profile (OHIP-14) ⁽¹⁴⁾, which was adapted and validated for the Brazilian Portuguese language ⁽¹⁵⁾. The OHIP-14 is an instrument that evaluates the OHRQoL by assessing the frequency of problems associated with the mouth in the last six months, in seven dimensions: functional limitation, physical pain, psychological discomfort, physical disability, psychological disability, social disability, and handicap. The answers to each question are obtained by choosing one option on a Likert scale: never=0 point, rarely=1 point, sometimes=2 points, often=3 points, or always=4 points. The sum of the answers provides a score ranging from 0 to 56 ⁽¹⁴⁾, the higher the score, the poorer the OHRQoL.

In the second stage of the study, three researchers (BSD, LDC and RKT) applied the OHIP-14 questionnaire via telephone interview. This call was made after a time interval of 15 days to avoid possible memory bias ⁽⁸⁾. Previous training to standardize the approach was carried out, and the researchers were instructed to just repeat the questions that were in the questionnaire, without any additional explanation.

Statistical analysis was performed using STATA 12.0 software (Stata Corporation, College Station, TX, USA). First, Shapiro-Wilk normality test was applied to investigate data distribution and then define the statistical tests to be used. As data was not normally distributed, a nonparametric test was chosen. Differences in the overall and domain-specific OHIP-14 scores between methods (OA or TA) were compared using the Wilcoxon rank test (α = 5%). Cronbach’s alpha coefficient ⁽¹⁶⁾ assessed the reliability between the overall and domain-specific OHIP-14 scores according to the method of administration. Intraclass Correlation Coefficient (ICC) assessed test-retest reliability, allowing us to assess the reproducibility of the questionnaires ⁽¹⁷⁾.

The Bland-Altman plot ⁽¹⁸⁾ was used to evaluate the correlation between the two methods of application, with its limit of agreement and confidence interval. Limits of agreement around the mean difference were calculated as 1.96 times the standard deviation (SD) of the differences. Thus, this statistic represents the test–retest differences expected for 95% of the individuals in the sample. Finally, a simple linear regression was performed to assess the presence of systematic bias ⁽¹⁹⁾. In this case, the mean of the methods was included as the independent variable and the difference between them as the dependent variable.

RESULTS

Of the1,197 eligible participants, 111 answered the first stage of the study (OA), and 109 answered the second stage (TA). Therefore, 109 18-22-year-old young adults with a mean age of 19 years (SD 1) were included in this study. As shown in Table 1, most of them were girls (68.8%) and had white skin color (73.8%).

Table 1. Sample characteristics of the study participants.
Variable	n (%)
Sex
Male	34 (31.2)
Female	75 (68.8)
Skin Color*
Non-White	28 (26.2)
White	79 (73.8)
Age
18	33 (30.3)
19	37 (33.9)
≥20	39 (35.8)
Level education
High school University/technical qualification	46 (42.2) 57 (52.3)
School evasion	6 (5.5)
Family income1
≤2 BMW	53 (52)
>2 BMW	49 (48)
Visit to a dentist (last 6 months)
No	65 (59.6)
Yes	44 (40.4)
Self-reported gingival bleeding
No	62 (56.9)
Yes	47 (43.1)
Tooth brushing frequency
≤twice/day	68 (62.4)
>twice/day	41 (37.6)
TOTAL	109 (100)
1Missing data BMW: Brazilian Minimum Wage

Table 2 presents total and domain-specific mean (SD) and median (interquartile range [IQR]) OHIP-14 scores according to the method of administration. While the mean and median OHIP-14 score were 9.72 (SD 9.71) and 6 (IQR 3; 14) for the OA, respectively, these figures were 8.38 (SD 8.43) and 6 (IQR 2; 11) for the TA. Wilcoxon rank test showed significant differences between methods for the overall score (p=0.02) and psychological disability domain (p<0.01), with the OA resulting in higher scores than the TA.

Table 2. Total and domain-specific mean and median OHIP-14 scores according to the method of administration.
	Online Administration			Telephone administration
Domain	Mean (SD)	Median (IQR)	Range	Mean (SD)	Median (IQR)	Range		p value*
Functional limitation	0.7 (1.3)	0 (0; 1)	0-7	0.8 (1.2)	0 (0; 1)	0-6		0.45
Physical pain	2.5 (1.8)	2 (1; 4)	0-8	2.5 (1.9)	2 (1; 4)	0-8		0.76
Psychological discomfort	2.6 (2.3)	2 (1; 4)	0-8	2.1 (2.1)	2 (0; 3)	0-8		<0.01
Physical disability	1.0 (1.6)	0 (0; 2)	0-6	0.8 (1.2)	0 (0; 1)	0-6		0.36
Psychological disability	1.6 (2.0)	1 (0; 2)	0-8	1.3 (1.8)	0 (0; 2)	0-8		0.20
Social disability	0.7 (1.6)	0 (0; 0)	0-6	0.6 (1.2)	0 (0; 1)	0-6		0.55
Handicap	0.6 (1.2)	0 (0; 0)	0-6	0.4 (0.9)	0 (0; 0)	0-4		0.20
Overall score	9.7 (9.7)	6 (3; 14)	0-48	8.4 (8.4)	6 (2; 11)	0-42		0.02
SD = Standard deviation; IQR = Interquartile range. *Wilcoxon rank test.

Cronbach's alpha for the overall OHIP-14 score was 0.92, ranging from 0.76 (social disability domain) to 0.90 (psychological disability domain), as shown in Table 3. ICC was 0.76 (overall score), ranging from 0.57 (psychological discomfort domain) to 0.77 (handicap domain).

Table 3. Reliability (Cronbach’s alpha), test-retest reliability (ICC) and mean differences between administration methods in the total domain-specific scores of OHIP-14.
Online-Telephone	Cronbach’s alpha	ICC (95%CI)	Mean difference (95%CI)
Functional limitation	0.86	0.76 (0.43 to 1.08)	-0.07 (-0.24 to 0.10)
Physical pain	0.89	0.70 (0.43 to 0.97)	0.06 (-0.15 to 0.28)
Psychological discomfort	0.82	0.57 (0.25 to 0.89)	0.55 (0.23 to 0.87)
Physical disability	0.82	0.65 (0.25 to 1.05)	0.19 (-0.02 to 0.41)
Psychological disability	0.90	0.76 (0.48 to 1.03)	0.99 (0.67 to 1.32)
Social disability	0.76	0.60 (0.16 to 1.04)	0.17 (-0.07 to 0.40)
Handicap	0.79	0.77 (0.44 to 1.11)	0.16 (-0.01 to 0.33)
Overall score	0.92	0.76 (0.62 to 0.91)	1.35 (0.43 to 2.27)

The Bland-Altman plot (Figure 1) showed that most differences between OA and TA concentrated within the limits of agreement, demonstrating a high correlation. The simple linear regression reached statistical significance (p=0.004), which points systematic bias with OA systematically tending to present higher scores than TA.

Figure 1. Bland-Altman plot considering OHIP-14 differences between online and telephone administration in the overall score and limits of concordance.

DISCUSSION

This study compared the TA of OHIP-14 questionnaire with the OA in young adults. In general, the method of administration did not influence the OHIP-14 score, as the ICC and the Cronbach’s alpha values confirmed that the responses were highly correlated. To the best of our knowledge, this study is the first to evaluate the digital self-administration of the OHIP-14 questionnaire.

As for reliability (Cronbach's alpha) and reproducibility (ICC), our study found satisfactory values for both parameters in the overall OHIP-14 score (0.92 and 0.76, respectively), such as in the studies that compared different methods of administration of OHIP-49 ^(9,10) and Early Childhood Oral Health Impact Scale (ECOHIS) ⁽¹¹⁾. Desai et al. ⁽¹⁰⁾ obtained a Cronbach's alpha of 0.97 and an ICC of 0.89 between telephone interview and self-administration of the English version of OHIP-49. Reissman et al. ⁽⁹⁾ examined three methods of administration of the German version of OHIP-49 (face-to-face interview, telephone interview and self-administered questionnaire) and found Cronbach's alpha between 0.96 and 0.97 and ICC between 0.88 and 0.93. Ortiz et al. ⁽¹¹⁾ compared the administration of ECOHIS by face-to-face interview and via telephone and found a Cronbach’s alpha of 0.96 and an ICC of 0.93 between these methods. The only study comparing different methods of administration of the Brazilian version of OHIP-14 (interview versus self-administered) total scores were not influenced by the method ⁽⁸⁾. On the other hand, Bekes et al. ⁽¹²⁾ comparing three methods of administration of the German version of ECOHIS in Austrian children showed satisfactory ICC values (>0.70) between self-administered versus telephone and between interview versus telephone, but ICC<0.70 for the comparison between self-administered versus interview. Although for ICC a minimum value of 0.70 is considered satisfactory, there is no consensus regarding reliability/internal consistency. Some studies define Cronbach's alpha >0.70 as satisfactory, while others consider values >0.60 ⁽²⁰⁾. The Cronbach’s alpha found in our study would be considered satisfactory according to these two different cutoff points. Corroborating the reliability tests, the Bland-Altman plot showed a high correlation between methods, as most of the differences were within the limits of agreement.

Our findings showed that young adults reported poorer OHRQoL in the overall OHIP-14 score (p=0.02) and psychological discomfort domain (p<0.01) when the method was OA, which is in agreement with the study by Desai et al. (2014) ⁽¹⁰⁾. The authors found significantly higher median overall OHIP-49 score when the questionnaire was self-administered when compared with telephone interview. This finding can be explained by the fact that patients may feel more comfortable on OA, without the presence of (and possible judgment from) an interviewer. Response bias influenced by the interview situation ⁽²¹⁾, mainly ‘socially desirable responding’, a response bias defined as the tendency of respondents to provide predominantly positive self-descriptions ⁽²²⁾, may be minimized in self-administered formats, yielding more candid responses. This hypothesis is related to the social stigma of talking openly about psychological questions, mainly for a non-familiar person. Supporting this reasoning, a previous study showed that web surveys might increase reports of sensitive information ⁽²³⁾. Furthermore, each participant answers the online questionnaire at the most opportune moment, when they have the time or desire to do so. On the other hand, the participant may receive the phone call at an inappropriate moment. In addition, full reading of answer options for multiple questions tends to turn the call boring for respondents and laborious for researchers (for example, in the present study, the options “never, rarely, sometimes, often, or always” had to be repeated 14 times during the call). In this sense, it was previously showed that aural mode respondents tend to give extreme responses on the positive sides of the scales compared to visual mode respondents ⁽²⁴⁾. As a disadvantage of OA, researchers have less control of missing data, which does not occur through the TA as participants usually answer all questions. This aspect has been previously shown by Sousa et al. ⁽⁸⁾, who observed higher values of completion in the interview method than self-administration. Another unfavorable circumstance may be the need to have a device connected to internet (smartphone, tablet, or computer) to answer online questionnaires. Although this was not a problem for our study sample, it may become a problem for elderly people or participants with lower socioeconomic status.

The lack of a third comparison group composed of face-to-face interview or personally self-administered may be seen as a limitation of this study, but it was not possible due to the restrictions imposed by the COVID-19 pandemic. At the time of data collection, face-to-face meetings were not recommended. Notwithstanding, both methods resemble those used in our study: the online questionnaire is self-administered, and the telephone interview has an interlocutor, as well as the face-to-face interview. Another possible limitation of the study is the lack of randomization to define the order of application. All participants completed the online questionnaire 15 days before the telephone interview. The absence of randomization may have introduced order bias (order effect), in which the experience of the first method influences the responses in the second. This type of bias occurs, for example, when participants remember their previous responses, adapt their behavior based on the first experience, or become more familiar with the instrument. Randomizing the order could have minimized this effect. As strengths, the interviewers underwent training before data collection, in which they were instructed to follow a protocol of just repeating the questions in cases of doubts, without giving any additional explanation. In addition, there was an interval of 15 days between the methods, in order to avoid memory bias ⁽⁸⁾.

In conclusion, this study showed that both methods of administration had high values of reliability and moderate values of reproducibility; however, the OA systematically resulted in higher scores than the TA. Therefore, it is suggested that the same method of administration of OHIP-14 should be used from the beginning to the end of a study. Future studies, including a third group with face-to-face interview, could add valuable knowledge to this research field.

ACKNOWLEDGEMENTS

We acknowledge the support provided by the Federal University of Santa Maria, and the National Coordination of Post-graduate Education (CAPES), Ministry of Education, Brazil (funding code 001). We also thank the Research Support Foundation of the State of Rio Grande do Sul (FAPERGS), National Council for Scientific and Technological Development (CNPq) and all the adolescents who took part in this study.

REFERENCES

1. Sischo L, Broder HL. Oral health-related quality of life: What, why, how, and future implications. J Dent Res 2011;90(11):1264–1270; doi: 10.1177/0022034511399918.

2. Bulgareli JV, Faria ET de, CortellazziI KL, et al. Fatores que influenciam o impacto da saúde bucal nas atividades diárias de adolescentes, adultos e idosos. Rev Saude Publica 2018;52:44; doi: 10.11606/s1518-8787.2018052000042.

3. Guerra MJC, Greco RM, Leite ICG, et al. Impact of oral health conditions on the quality of life of workers. Ciencia e Saude Coletiva 2014;19(12):4777–4786; doi: 10.1590/1413-812320141912.21352013.

4. World Health Organization. Novel Coronavirus(2019-nCoV) Situation Report – 22- 11 February 2020. World Health Organization 2020;(February):2019.

5. Tysiąc-Miśta M, Dziedzic A. The attitudes and professional approaches of dental practitioners during the COVID-19 outbreak in poland: A cross-sectional survey. Int J Environ Res Public Health 2020;17(13):1–17; doi: 10.3390/ijerph17134703.

6. Aliaño Piña M, Ruiz Villén C, Galán Serrano J, et al. Cardiopulmonary resuscitation during the COVID-19 pandemic in Spain. Revista Española de Anestesiología y Reanimación (English Edition) 2021;68(8):437–442; doi: 10.1016/j.redare.2021.09.001.

7. Kwas K, Nowakowska A, Fornalczyk A, et al. Impact of contraception on uterine fibroids. Medicina (Lithuania) 2021;57(7); doi: 10.3390/medicina57070717.

8. de Sousa PCB, Mendes FM, Pettorossi Imparato JC, et al. Differences in responses to the Oral health impact profile (OHIP14) used as a questionnaire or in an interview. Braz Oral Res 2009;23(4):358–364; doi: 10.1590/s1806-83242009000400002.

9. Reissmann DR, John MT, Schierz O. Influence of administration method on oral health-related quality of life assessment using the Oral Health Impact Profile. Eur J Oral Sci 2011;119(1):73–78; doi: 10.1111/j.1600-0722.2010.00805.x.

10. Desai R, Durham J, Wassell RW, et al. Does the mode of administration of the Oral Health Impact Profile-49 affect the outcome score? J Dent 2014;42(1):84–89; doi: 10.1016/j.jdent.2013.10.016.

11. Ortiz FR, dos Santos MD, Landenberger T, et al. Comparison of face-to-face interview and telephone methods of administration on the ECOHIS scores. Braz Dent J 2016;27(5):613–618; doi: 10.1590/0103-6440201601134.

12. Bekes K, Solanke C, Waldhart T, Priller J, Stamm T. Effect of method of administration on the oral health-related quality of life assessment using the Early Childhood Oral Health Impact Scale (ECOHIS-G). Clin Oral Investig. 2021;25(8):5061-5066. doi:10.1007/s00784-021-03818-7.

13. Racki DNDO, Nora ÂD, Comim LD, et al. Erosive tooth wear among South Brazilian adolescents, and its association with sociodemographic variables. Braz Oral Res 2019;33:1–11; doi: 10.1590/1807-3107bor-2019.vol33.0119.

14. Slade GD. Derivation and validation of a short-form oral health impact profile. Community Dent Oral Epidemiol 1997;25(4):284–290; doi: 10.1111/j.1600-0528-1997.tb00941.

15. de Oliveira BH, Nadanovsky P. Psychometric properties of the Brazilian version of the Oral Health Impact Profile - Short form. Community Dent Oral Epidemiol 2005;33(4):307–314; doi: 10.1111/j.1600-0528.2005.00225.

16. Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika 1951;16(3):297–334; doi: 10.1007/BF02310555.

17.Schuck P. Assessing reproducibility for interval data in health-related quality of life questionnaires: Which coefficient should be used? Quality of Life Research 2004;13(3):571–585; doi: 10.1023/B:QURE.0000021318.92272.2a.

18. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999;8(2):135–160; doi: 10.1177/096228029900800204.

19. Lyons RA, Wareham K, Lucas M, et al. SF-36 scores vary by method of administration: implications for study design. J Public Health (Bangkok) 1999;21(1):41–45; doi: https://doi.org/10.1093/pubmed/21.1.41.

20. Souza AC de, Alexandre NMC, Guirardello E de B. Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade. Epidemiol Serv Saude 2017;26(3):649–659; doi: 10.5123/S1679-49742017000300022.

21. Bogner, K., & Landrock, U. GESIS Survey Guidelines Response Biases in Standardised Surveys. 2016. 10.15465/gesis-sg_en_016.

22. Paulhus, D. L. Socially desirable responding: The evolution of a construct. In H. I. Braun, D. N. Jackson, & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (pp. 49-69). 2002; Mahwah NY: Erlbaum.

23. Kreuter F, Presser S, Tourangeau R. Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opin Q 2008;72(5):847–865; doi: 10.1093/poq/nfn063.

24. Dillman DA, Phelps G, Tortora R, et al. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the Internet. Soc Sci Res 2009;38(1):1–18; doi: 10.1016/j.ssresearch.2008.03.007.