Psychometric Properties of Self-report Questionnaires in Evaluating Blended Learning in Health Science University Students: A Systematic Review

Abstract

Background. Due to the COVID-19 outbreak, schools had to switch online. The sudden transition to blended teaching and learning (BTL) poses challenges for students and teachers, especially for health science programs that require hands-on practical experience. The validity, reliability, and responsiveness of these self-report questionnaires (SRQs) should be established to ensure the accuracy of the results as intended by the SRQ.

Objectives. This study critically appraised, compared, and summarized the psychometric properties of SRQ evaluating BTL among health science university students. This review determined the SRQ’s reliability, internal consistency, various forms of validity (content, criterion, construct), and responsiveness.

Methods. Following a 10-step procedure based on COSMIN guidelines, we conducted a systematic review of SRQs used by health science university students to evaluate blended teaching and learning. Studies were eligible if they reported psychometric properties of SRQs related to blended learning among university health science students; exclusions included studies focusing on perceptions, attitudes, self-efficacy, and satisfaction, as well as articles such as biographies, editorials, and conference materials. Searches covered multiple electronic databases until April 26, 2023, including PubMed, EMBASE, Web of Science, MEDLINE (OVID), PsycInfo, CINAHL, EBSCOHOST, ERIC, Scopus, Science Direct, Google Scholar, JSTOR, Acta Medica Philippina, Philippine Journal of Health Research and Development, and HERDIN, managed through Zotero. Two independent reviewers performed database searches, title and abstract screening, and full-text evaluations, with a third reviewer resolving any disputes. The COSMIN Risk of Bias Checklist was employed to evaluate included studies on the development and various measurement properties of SRQs. The reviewers assessed SRQ standards, including validity, reliability, internal consistency, measurement error, responsiveness, interpretability, and feasibility. Data extraction and result tabulation were independently completed, with content comparison by two health education experts. This evaluation categorized the SRQs into three quality and validity levels.

Results. The study examined five articles; four were rated as 'doubtful' and one as 'inadequate' in the overall development of SRQ. All four 'doubtful' studies demonstrated questionable content validity when university students were asked about the questionnaire's relevance, comprehensiveness, and comprehensibility. Only half of these studies achieved an 'adequate' rating for content validity based on expert opinions on relevance and comprehensiveness. All but one study scored from 'very good' to 'adequate' in structural validity. Three out of the four studies scored a very good rating for internal consistency, while one was deemed 'inadequate' in internal consistency, cross-cultural validity, and reliability. Three out of four studies scored 'very good' on construct validity, but all overlooked criterion validity and responsiveness. Conducted in various locations, including Australia, Romania, Turkey, and Taiwan, these studies highlighted both common characteristics and limitations in questionnaire development according to the COSMIN guidelines. Four studies were deemed reliable and valid for BTL constructs (Category A); Wu et al. requires further validation (Category B). Study limitations included heterogeneity in populations, settings, and questionnaire versions, potential subjective bias in SRQ content comparison, and the evolving nature of SRQs in blended learning contexts.

Conclusion. The systematic review reports the development and evaluation of SRQs for BTL while identifying gaps in their applicability to health science programs. The Blended Learning Scale (BLS) of Lazar et al. and the Blended Learning Questionnaire (BLQ) of Ballouk et al. showed an ‘adequate' rating for content validity. BLS revealed very good structural validity, internal consistency, and adequate content validation. Although the BLQ lacked Confirmatory Factor Analysis, it yielded valuable constructs for evaluating health sciences students' experiences in BTL. Both tools require improvements on recall period, completion time, interpretability, and feasibility. The review underscores the necessity for cont inuous assessment and enhancement of such instruments in BTL, advocating a rigorous scale development process. Furthermore, it encourages the customization of teaching and learning evaluation tools to suit specific institutional contexts while promoting further validation of these questionnaires across different populations in future research.