Content Validity and Reliability Analysis of Instruments on Students’ Perceptions of Flipped STEM Learning
DOI:
https://doi.org/10.11113/sh.v18n2.2284Keywords:
Flipped classroom, STEM, CVI analysis, reliabilityAbstract
As the flipped classroom model continues to gain attention in secondary school STEM classrooms and educational quantitative research focusing on instrument development and validation, the need for a valid and reliable measurement instrument that is appropriate for the Malaysian secondary school context has become increasingly important. In STEM education, researchers seek to better understand students’ learning experiences within flipped classroom practices within STEM-specific learning environments. Thus, this study aimed to investigate validity and reliability of the student perception of flipped classroom (SPFC) instrument; which measures; (i) collaborative environment, (ii) self-directed learning, (iii) learning impact, (iv) motivation and enjoyment, and (v) technology integration. Content validity was evaluated by ten experts through the application of the item-level content validity index (I-CVI), scale-level content validity index (S-CVI), and Kappa analysis. The study findings showed that the SPFC instrument demonstrated strong content validity (S-CVI/Ave = 0.96; S-CVI/UA = 0.85). Seventeen (17) items achieved full agreement (I-CVI = 1.00), while three items with lower CVI values were retained after clarity-focused revisions. A pilot study involving 30 students was conducted to assess reliability using Cronbach’s alpha. The overall reliability was excellent (α = .927), with construct-level alphas ranging from acceptable to strong: collaborative environment (α = .866), self-directed learning (α = .685), learning impact (α = .873), motivation and enjoyment (α = .720), and technology integration (α = .884). In conclusion, the 20 items of SPFC questionnaire were accepted and retained to better understand students' perceptions of flipped classroom learning experiences in STEM context. Future research is recommended to further examine the instrument’s construct validity and applicability across different STEM subjects and educational levels.
References
Abdul Mutalib, E., & Mohd Zulkanain, M. (2024). Sorotan kemerosotan mata pelajaran Sains dan Matematik: Fenomena biasa pendidikan negara? Astroawani.com. https://www.astroawani.com/berita malaysia/kolumnis sorotan kemerosotan mata pelajaran sains dan matematik fenomena biasa pendidikan negara 475930 retrieved on 23 June 2024
Aburayash, H. (2021). The students attitudes’ toward the flipped classroom strategy and relationship to self-learning skills. Journal of Education and Learning (EduLearn), 15(3). DOI: https://doi.org/10.11591/edulearn.v15i3.18132
Agyeman, N. Y. B., & Aphane, V. (2024). Implementing Flipped Classroom to Enhance Student Engagement: An Action Research. E-Journal of Humanities, Arts and Social Sciences, 5(11), 1860–1878. DOI: https://doi.org/10.38159/ehass.202451119
Aspin, S. H., Ali, M., Abd, M., & Bunyamin, H. (2022). STEM Education in Malaysia: A Review. Learning Science And Mathematics Journal, 1(17), 125–139. DOI: https://doi.org/2637-0832
Bahari, F. A. Z., & Saleh, S. (2023). Content Validation Procedure: Development of Problem-solving Skills Test (PSST). Jurnal Pendidikan Sains Dan Matemaitk Malaysia, 13(1), 1–9. DOI: https://doi.org/10.37134/jpsmm.vol13.1.1.2023
Bishop, J. L., & Verleger, M. A. (2013). The Flipped Classroom: A Survey of the Research. 120th American Society for Engineering Education Annual Conference and Exposition, 30, 1-18.
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6(149). DOI: https://doi.org/10.3389/fpubh.2018.00149
Bogaard, G., Dimopoulos, R., & Quaedflieg, C. W. E. M. (2025). Lying Under Pressure: Examining the Impact of Stress and Veracity on Verbal and Nonverbal Behaviors in Credibility Judgments. Applied Cognitive Psychology, 39(1). DOI: https://doi.org/10.1002/acp.70018
Cameron, B. T., & Hynes, C. (2024). Expert Panels in Evaluation: An Update From the Field Using the DATA Model. Canadian Journal of Program Evaluation, 39(1), 117–143. DOI: https://doi.org/10.3138/cjpe-2023-0047
Cheah, S. M., & Sale, D. (2019). Enhancing Student Engagement In Flipped Classroom Using Autonomy-Supportive Teaching. Proceedings of the International CDIO Conference.
Cho, H. J., Zhao, K., Lee, C. R., Runshe, D., & Krousgrill, C. (2021). Active learning through flipped classroom in mechanical engineering: improving students’ perception of learning and performance. International Journal of STEM Education, 8(1). DOI: https://doi.org/10.1186/s40594-021-00302-2
Chung, E. J., & Lee, B.-H. (2018). The effects of flipped learning on learning motivation and attitudes in a class of college physical therapy students. Journal of Problem-Based Learning, 5(1), 29-36. DOI: https://doi.org/10.24313/jpbl.2018.5.1.29
Creswell, J. W. ., & Creswell, J. David. (2022). Research Design: Qualitative, Quantitative, and Mixed Methods Approaches (6th ed.). SAGE Publications, Inc.
Davis, L. L. (1992). Instrument review: Getting the most from a panel of experts. Applied Nursing Research, 5(4), 194-197. DOI: https://doi.org/10.1016/S0897-1897(05)80008-4
Devellis, R. F. (2016). Scale Development : Theory and Applications (4th ed.). SAGE Publications.
Divjak, B., Rienties, B., Iniesto, F., Vondra, P., & Žižak, M. (2022). Flipped classrooms in higher education during the COVID-19 pandemic: findings and future research recommendations. In International Journal of Educational Technology in Higher Education. 19(1). DOI: https://doi.org/10.1186/s41239-021-00316-4
Ekinci, F., Bektaş, O., Karaca, M., & Yiğit, K. N. (2023). The flipped learning perception scale: A validity and reliability study. Education and Information Technologies, 28(11), 14139–14166. DOI: https://doi.org/10.1007/s10639-023-11731-7
Elian, S. A., & Hamaidi, D. A. (2018). The effect of using flipped classroom strategy on the academic achievement of fourth grade students in Jordan. International Journal of Emerging Technologies in Learning, 13(2), 110–125. DOI: https://doi.org/10.3991/ijet.v13i02.7816
Environmental Association for Colleges and Universities (EAUC)). (2025). SDG Accord Report 2025. sdgaccord.org
Eren, E. U., & Kenan Dikilitaş. (2025). Teachers’ motivational strategies and student motivation across teaching modalities. Interactive Learning Environments, 1–14. DOI: https://doi.org/10.1080/10494820.2025.2601300
Fairhurst, N., Koul, R., & Sheffield, R. (2023). Students’ perceptions of their STEM learning environment. Learning Environments Research, 26(3). DOI: https://doi.org/10.1007/s10984-023-09463-z
Feijóo, J. C. M., Suárez, F., Chiyón, I., & Alberti, M. G. (2021). Some web-based experiences from flipped classroom techniques in aec modules during the covid-19 lockdown. Education Sciences, 11(5). DOI: https://doi.org/10.3390/educsci11050211
George, D., & Mallery, P. (2019). Reliability Analysis. IBM SPSS Statistics 25 Step by Step: A Simple Guide And Reference. In IBM SPSS Statistics 25 Step by Step.
Gong, J., Cai, S., & Cheng, M. (2024). Exploring the Effectiveness of Flipped Classroom on STEM Student Achievement: A Meta-analysis. Technology, Knowledge and Learning, 29(2). DOI: https://doi.org/10.1007/s10758-023-09700-7
Hakim Mahari, & Qistina Sallehuddin. (2025). Nearly half of Form Four students taking STEM subjects. NST Online; New Straits Times. https://www.nst.com.my/news/nation/2025/10/1300142/nearly-half-form-four-students-taking-stem-subjects retrieved on 23 June 2024
Hertzog, M. A. (2008). Considerations in Determining Sample Size for Pilot Studies. Research in Nursing & Health, 31(2), 180–191. DOI: https://doi.org/10.1002/nur.20247
Intan, S. N. A., & Yasin, S. M. (2021). Content Validity of a Self-Reported Instrument for Safety and Health (S&H) Culture Practice In Paramedic Training Institute Using A Heterogeneous Expert Panel. Turkish Journal of Computer and Mathematics Education (TURCOMAT), 12(7), 2464–2472. https://turcomat.org/index.php/turkbilmat/article/view/3574/3052
Ismail, M. H. Bin, Salleh, M. F. M., & Nasir, N. A. M. (2019). The Issues and Challenges in Empowering STEM on Science Teachers in Malaysian Secondary Schools. International Journal of Academic Research in Business and Social Sciences, 9(13), 430-444. DOI: https://doi.org/10.6007/IJARBSS/v9-i13/6869
Jamaluddin, F., Razak, A. Z. A., & Rahim, S. S. A. (2025). Navigating the challenges and future pathways of STEM education in Asia-Pacific region: A comprehensive scoping review. STEM Education, 5(1), 53–88. DOI: https://doi.org/10.3934/steme.2025004
Johanson, G. A., & Brooks, G. P. (2010). Initial Scale Development: Sample Size for Pilot Studies. Educational and Psychological Measurement, 70(3), 394–400. DOI: https://doi.org/10.1177/0013164409355692
Kalithas, M., & Manivannan, M. L. (2024). UKM TESL Students’ Perceptions and Satisfaction on the Use of Flipped Classroom Approach. LSP International Journal, 11(2), 115–125. DOI: https://doi.org/10.11113/lspi.v11.22901
Karadag, R., & Keskin, S. S. (2017). The effects of flipped learning approach on the academic achievement and attitudes of the students. New Trends and Issues Proceedings on Humanities and Social Sciences, 4(6). DOI: https://doi.org/10.18844/prosoc.v4i6.2926
Karjanto, N., & Acelajado, M. J. (2022). Sustainable Learning, Cognitive Gains, and Improved Attitudes in College Algebra Flipped Classrooms. Sustainability (Switzerland), 14(19). DOI: https://doi.org/10.3390/su141912500
Karnia, R. (2024). Importance of Reliability and Validity in Research. Psychology and Behavioral Sciences, 13(6), 137–141. https://doi.org/10.11648/j.pbs.20241306.11
Kay, R., & Dermott, K. (2018). Examining Stem-Based Flipped Classrooms in Higher Education: A Review of the Literature. ICERI2018 Proceedings, 1. DOI: https://doi.org/10.21125/iceri.2018.0076
Kozikoğlu, İ., Erbenzer, E., & Ateş, G. (2021). The Relationship Between Teachers’ Perceptions of Flipped Learning Self-efficacy and Their Behaviors to Support Learner Autonomy. Atatürk Üniversitesi Kazım Karabekir Eğitim Fakültesi Dergisi, 42, 344–366. DOI: https://doi.org/10.33418/ataunikkefd.796531
Lam, C. P., & Siew, N. M. (2024). Flipped Classroom in Science Education: Correlating Student Experience with Attitudes. Problems of Education in the 21st Century, 82(5), 672–686. https://doi.org/10.33225/pec/24.82.672 Lamm, K., Lamm, A., & Edgar, D. (2020). Scale Development and Validation: Methodology and Recommendations. Journal of International Agricultural and Extension Education, 27(2), 24–35. DOI: https://doi.org/10.5191/jiaee.2020.27224
Lo, C. K., & Hew, K. F. (2019). The impact of flipped classrooms on student achievement in engineering education: A meta-analysis of 10 years of research. Journal of Engineering Education, 108(4), 523-546. DOI: https://doi.org/10.1002/jee.20293
Lukmanul Hakim, N. A. M., Rasidi Pairan, M., & Ikram Zakaria, M. (2025). Step-By-Step Guide to Calculating Content Validity Index (CVI) For Single Constructs Using Excel. International Journal of Research and Innovation in Social Science, IX(III), 1717–1726. https://doi.org/10.47772/ijriss.2025.90300135
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6). DOI: https://doi.org/10.1097/00006199-198611000-00017
Ma, Y. (2023). Exploration of flipped classroom approach to enhance critical thinking skills. Heliyon, 9(11), e20895. DOI: https://doi.org/10.1016/j.heliyon.2023.e20895
Martínez-Jiménez, R., & Ruiz-Jiménez, M. C. (2020). Improving students’ satisfaction and learning performance using flipped classroom. International Journal of Management Education, 18(3). DOI: https://doi.org/10.1016/j.ijme.2020.100422
Mat Said, N. A., Bujang, S. M., Buang, N. A., Siraj@Ramli, H. H., & Awang Besar, M. N. (2022). Critical Thinking Transfer Practice Instrument: A Content Validity Calculation Steps Based on Expert Panel Evaluation. Education in Medicine Journal, 14, 61–74. DOI: https://doi.org/10.21315/eimj2022.14.3.5
Md Zainuddin, M. Z. A., Mohd Noor, N. H., Johari, N. ’Aisyah, Othman, E. S., & Mat Fadzil, A. F. (2024). Flipped Classroom and its Effects on Student’s Academic Performance and Behaviour. International Journal of Academic Research in Business and Social Sciences, 14(9), 399-415. DOI: https://doi.org/10.6007/IJARBSS/v14-i9/22372
Milligan, M. (2022, September 1). Why are the UN Sustainable Development Goals Important to STEM? Medium. https://abetaccredit.medium.com/why-are-the-un-sustainable-development-goals-important-to-stem-e08a6addfceb
Nja, C. O., Orim, R. E., Neji, H. A., Ukwetang, J. O., Uwe, U. E., & Ideba, M. A. (2022). Students’ attitude and academic achievement in a flipped classroom. Heliyon, 8(1). DOI: https://doi.org/10.1016/j.heliyon.2022.e08792
Nolan, E., Brady, M., Rienties, B., & Héliot, Y. (2021). Once more on the rollercoaster: Loses and gains from the rapid shift to online delivery during Covid. Academy of Management Proceedings, 2021(1). DOI: https://doi.org/10.5465/ambpp.2021.15358abstract
Nunnally, J., & Bernstein, I. (1994). Psychometric Theory, 3rd edn, 1994. McGraw-Hill, New York, 3.
Öztürk, M., & Çakıroğlu, Ü. (2021). Flipped learning design in EFL classrooms: implementing self-regulated learning strategies to develop language skills. Smart Learning Environments, 8(1). DOI: https://doi.org/10.1186/s40561-021-00146-x
Pallant, Julie. (2020). SPSS Survival Manual: A Step by Step Guide to Data Analysis Using IBM SPSS (7th ed.). Routledge: Allen & Unwin. https://www.taylorfrancis.com/books/mono/10.4324/9781003117452/spss-survival-manual-julie-pallant
Pilu, R., Jabu, B., & Sulaiman, I. (2025). Flipped learning and its challenges: understanding students’ struggles in Indonesian EFL contexts from teachers’ viewpoint. Frontiers in Education, 10,. DOI: https://doi.org/10.3389/feduc.2025.1575385
Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Research in Nursing and Health, 29(5), 489-497. DOI: https://doi.org/10.1002/nur.20147
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Focus on research methods: Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health, 30(4), 459-467.. DOI: https://doi.org/10.1002/nur.20199
Rahman, L. (2024). Pedagogical Approach in STEM Education: A Literature Review. International Journal of Engineering Research & Technology (IJERT), 13(9). DOI: https://doi.org/10.5281/zenodo.18137298
Rodrigues, I. B., Adachi, J. D., Beattie, K. A., & MacDermid, J. C. (2017). Development and validation of a new tool to measure the facilitators, barriers and preferences to exercise in people with osteoporosis. BMC Musculoskeletal Disorders, 18(1), 540. DOI: https://doi.org/10.1186/s12891-017-1914-5
Rubio, D. M., Berg-weger, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003). Objectifying Content Validity: In Social Work Research. Social Work Research, 27(2).
Sağlam, D., & Arslan, A. (2018). The Effect of Flipped Classroom on the Academic Achievement and Attitude of Higher Education Students. World Journal of Education, 8(4), 170-176. DOI: https://doi.org/10.5430/wje.v8n4p170
Segal, D. L., & Coolidge, F. L. (2018). Reliability. In M. H. Bornstein (Ed.), The SAGE Encyclopedia of Lifespan Human Development. SAGE Publications, Inc. DOI: https://doi.org/10.4135/9781506307633.n683
Shi, J., Mo, X., & Sun, Z. (2012). Content validity index in scale development. Journal of Central South University (Medical Sciences), 37(2), 152-155.. DOI: https://doi.org/10.3969/j.issn.1672-7347.2012.02.007
Shrotryia, V. K., & Dhanda, U. (2019). Content Validity of Assessment Instrument for Employee Engagement. SAGE Open, 9(1). 215824401882175. DOI: https://doi.org/10.1177/2158244018821751
Stefana, A., Damiani, S., Umberto Granziol, Umberto Provenzani, Solmi, M., Youngstrom, E. A., & Paolo Fusar-Poli. (2025). Psychological, psychiatric, and behavioral sciences measurement scales: best practice guidelines for their development and validation. Frontiers in Psychology, 15. DOI: https://doi.org/10.3389/fpsyg.2024.1494261
Sukserm, P. (2024). Determining the Appropriate Sample Size in EFL Pilot Studies. Journal of Research Methodology, 37(3), 245-264. https://doi.org/10.14456/jrm.2024.13
Syaiful, I. A., & Roebianto, A. (2020). Adapting and Examining the Factor Structure of the Self-Compassion Scale in Indonesian Version. Jurnal Psikologi, 47(3), 175-205. DOI: https://doi.org/10.22146/jpsi.57608
Syakdiyah, H., Wibawa, B., & Muchtar, H. (2018). The effectiveness of flipped classroom in high school Chemistry Education. IOP Conference Series: Materials Science and Engineering, 434(1). DOI: https://doi.org/10.1088/1757-899X/434/1/012098
Talan, T., & Gulsecen, S. (2019). The effect of a flipped classroom on students’ achievements, academic engagement and satisfaction levels. Turkish Online Journal of Distance Education, 20(4), 31 - 60. DOI: https://doi.org/10.17718/TOJDE.640503
Unal, A., & Unal, Z. (2021). Using Flipped Classroom in Middle Schools: Teachers’ Perceptions. Journal of Research in Education, 30(2), 90-112.
UNESCO. (2017). Education for Sustainable Development Goals: learning objectives. UNESCO. DOI: https://doi.org/10.54675/CGBA9153
Warmoth, K., O’Brien, L., Bilciu, A.-M., Dearling, J., Lynch, J., Woolham, J., Mioshi, E., & Almack, K. (2025). The role and contribution of experts by experience in building research capacity in adult social care services: findings from the script study. Research Involvement and Engagement, 11(1), 103. DOI: https://doi.org/10.1186/s40900-025-00779-z
Webb, M., & Doman, E. (2020). Impacts of flipped classrooms on learner attitudes towards technology-enhanced language learning. Computer Assisted Language Learning, 33(3), 240-274.. DOI: https://doi.org/10.1080/09588221.2018.1557692
Xiaoying, L., & Abu Samah, N. (2024). The Effectiveness, Benefits and Challenges of the Implementation of Flipped Classroom in English Teaching and Learning: A Systematic Review. Sains Humanika, 16(3), 113–121. DOI: https://doi.org/10.11113/sh.v16n3.2164
Yoong, S. M., Hosshan, H., Arumugam, S., Alya Qasdina Ng Ai Lee, Lau, S. C., & Govindasamy, P. (2022). Validity and Reliability of Needs Analysis Questionnaire for Dyscalculia Instrument. South Asian Journal of Social Science and Humanities, 3(3), 111-124. DOI: https://doi.org/10.48165/sajssh.2022.3307
Yusoff, M. S. B. (2019). ABC of Content Validation and Content Validity Index Calculation. Education in Medicine Journal, 11(2), 49-54. DOI: https://doi.org/10.21315/eimj2019.11.2.6
