Validity of Kirkpatrick Evaluation Model Instrument for Drug Prevention Education Programs in Primary School
DOI:
https://doi.org/10.24036/6Keywords:
Confirmatory Factor Analysis, Kirkpatrick Model, Exploratory Factor AnalysisAbstract
One of the evaluation models generally used in prevention programs is the Kirkpatrick evaluation. The model is used to escort the evaluation program closer to its effectiveness. Kirkpatrick's framework consists of four levels: reaction, learning, behaviour, and result. The purpose of the current research was to assess the validity of the Kirkpatrick evaluation model instrument for drug prevention programs in primary school students in Malaysia. This study used a survey research design by involving 692 primary school students in Malaysia. Three procedures were used to analyse the data in this research, namely Exploratory Factor Analysis (EFA), Confirmatory Factor Analysis (CFA) and Cronbach Alpha. EFA revealed the structure of the Kirkpatrick evaluation model which the reaction level has 12 items, the learning level has 8 items, the behaviour level has 6 items, and the result level has 6 items. At the same time, the CFA results showed that the model fit indices established a four-factor structure. Finally, the evaluation model has Cronbach's alpha value of .952, which exceeds the standard (.70 or above). It can be concluded that the Kirkpatrick evaluation model instrument was acceptable and reliable to assess the level of drug prevention programs among primary school students in Malaysia.
References
Adams, D., Chuah, K. M., Sumintono, B., & Mohamed, A. (2021). Students’ readiness for e-learning during the COVID-19 pandemic in a South-East Asian university: A Rasch analysis. Asian Education and Development Studies, 11(2), 324–339. https://doi.org/10.1108/aeds-05-2020-0100
Alavi, M., Visentin, D. C., Thapa, D. K., Hunt, G. E., Watson, R., & Cleary, M. (2020). Chi‐square for model fit in confirmatory factor analysis. Journal of Advanced Nursing, 76(9), 2209–2211. https://doi.org/10.1111/jan.14399
Alhadabi, A., & Aldhafri, S. (2021). A Rasch Model Analysis of the Psychometric Properties of the Student-Teacher Relationship Scale among Middle School Students. European Journal of Educational Research, 10(2), 957–973. https://doi.org/10.12973/eu-jer.10.2.957
Alsalamah, A., & Callinan, C. (2021). Adaptation of Kirkpatrick’s Four-Level Model of Training Criteria to Evaluate Training Programmes for Head Teachers. Education Sciences, 11(3), 116. https://doi.org/10.3390/educsci11030116
Anggraini, P. P., Apriliani, N. A., Supeni, I., & Handrianto, C. (2022). The use of the cocomelon youtube channel as a medium for introducing children's english vocabulary. SAGA: Journal of English Language Teaching and Applied Linguistics, 3(2), 81-90. https://doi.org/10.21460/saga.2022.32.137
Anwar, M. (2018). Business model innovation and smes performance: Does competitive advantage mediate? International Journal of Innovation Management, 22(07), 1850057. https://doi.org/10.1142/s1363919618500573
Arwin, A., Kenedi, A. K., Anita, Y., & Handrianto, C. (2022, June). The design of covid-19 disaster mitigation e-module for students of grades 1 in primary school. In 6th International Conference of Early Childhood Education (ICECE-6 2021) (pp. 173-176). Atlantis Press. https://doi.org/10.2991/assehr.k.220602.036
Aryadoust, V. (2016). Adapting Levels 1 and 2 of Kirkpatrick’s model of training evaluation to examine the effectiveness of a tertiary-level writing course. Pedagogies: An International Journal, 12(2), 151–179. https://doi.org/10.1080/1554480x.2016.1242426
Campbell, N. C., Murray, E., Darbyshire, J., Emery, J., Farmer, A., Griffiths, F., Guthrie, B., Lester, H., Wilson, P., & Kinmonth, A. L. (2007). Designing and evaluating complex interventions to improve health care. BMJ, 334(7591), 455–459. https://doi.org/10.1136/bmj.39108.379965.be
Chandrasekaran, R., Radzi, S., Kai, P. Z., Rajalingam, P., Rotgans, J., & Mogali, S. R. (2022). A validated instrument measuring students’ perceptions on plastinated and three‐dimensional printed anatomy tools. Anatomical Sciences Education, 15(5), 850–862. https://doi.org/10.1002/ase.2147
Dalimunthe, M. B. (2022). Kirkpatrick Four-level Model Evaluation: An Evaluation Scale on the Preservice Teacher’s Internship Program. Journal of Education Research and Evaluation, 6(2), 367–376. https://doi.org/10.23887/jere.v6i2.43535
Department of Statistics Malaysia Official Portal. (2022). https://www.dosm.gov.my/v1/
Drachsler, H., & Kirschner, P. A. (2012). Learner Characteristics. Encyclopedia of the Sciences of Learning, 1743–1745. https://doi.org/10.1007/978-1-4419-1428-6_347
Du, P., Liu, X., Zhong, G., Zhou, Z., Thomes, M. W., Lee, C. W., Bong, C. W., Zhang, X., Hao, F., Li, X., Zhang, G., & Thai, P. K. (2020). Monitoring Consumption of Common Illicit Drugs in Kuala Lumpur, Malaysia, by Wastewater-Cased Epidemiology. International Journal of Environmental Research and Public Health, 17(3), 889. https://doi.org/10.3390/ijerph17030889
Ghazali, M. A. I. M., Ali, Z., & Rani, A. A. (2019). Treating the addicted: cure and care service center (ccsc) in the east coast of Malaysia. International Journal of Humanities Technology and Civilization, 2(1), 1–12. https://journal.ump.edu.my/ijhtc/article/download/3446/552
Govindasamy, P., Olmos, A., Green, K., & Carmen Salazar, M. D. (2018). Application of Many Faceted Rasch Measurement with FACETS. Asian Journal of Assessment in Teaching and Learning, 8(1), 23–35. https://doi.org/10.37134/ajatel.vol8.3
Greer, A., Selfridge, M., Card, K., Benoit, C., Jansson, M., Lee, Z., & Macdonald, S. (2022). Factors contributing to frequent police contact among young people: A multivariate analysis including homelessness, community visibility, and drug use in British Columbia, Canada. Drugs: Education, Prevention and Policy, 29(2), 168-174.
Harrington, D. (2009). Confirmatory Factor Analysis. Oxford University Press.
Howard, M. C. (2015). A Review of Exploratory Factor Analysis Decisions and Overview of Current Practices: What We Are Doing and How Can We Improve? International Journal of Human-Computer Interaction, 32(1), 51–62. https://doi.org/10.1080/10447318.2015.1087664
Ingersoll, M., Hirschkorn, M., Landine, J., & Sears, A. (2018). Recruiting international educators in a global teacher shortage: Research for practice. The International Schools Journal, 37(2), 92-102.
Jackson, D. L., Gillaspy, J. A., & Purc-Stephenson, R. (2009). Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods, 14(1), 6–23. https://doi.org/10.1037/a0014694
Kamarulzaman, A. (2009). Impact of HIV prevention programs on drug users in Malaysia. JAIDS Journal of Acquired Immune Deficiency Syndromes, 52, S17-S19.
Khairunnisa, N., Rahman, M., & Handrianto, C. (2022). English digital literacy practices inside and outside class to develop students’ speaking skills. Pedagogy: Journal of English Language Teaching, 10(1), 13-24. https://doi.org/10.32332/joelt.v10i1.3790
Kirkpatrick, W. K. (2016). Kirkpatrick’s four levels of training evaluation. Amsterdam University Press.
Koning, I. M., De Kock, C., Van der Kreeft, P., Percy, A., Sanchez, Z. M., & Burkhart, G. (2021). Implementation of the Icelandic Prevention Model: A critical discussion of its worldwide transferability. Drugs: Education, Prevention and Policy, 28(4), 367-378.
Larsen, R., & Warne, R. T. (2010). Estimating confidence intervals for eigenvalues in exploratory factor analysis. Behavior Research Methods, 42(3), 871–876. https://doi.org/10.3758/brm.42.3.871
Livingston, W., Madoc-Jones, I., & Perkins, A. (2020). The potential of contribution analysis to alcohol and drug policy strategy evaluation: an applied example from Wales. Drugs: Education, Prevention and Policy, 27(3), 183-190.
Mahajan, M., & Singh, M. K. S. (2017). Importance and Benefits of Learning Outcomes. IOSR Journal of Humanities and Social Science, 22(03), 65–67. https://doi.org/10.9790/0837-2203056567
Miller, B. J. (2017). Utilising the Kirkpatrick model to evaluate a collegiate highimpact leadership development program [Thesis]. Texas A & M University.
Mindrila, D. (2017). Exploratory Factor Analysis: Applications in School Improvement Research (Education in a Competitive and Globalizing World). Nova Science Publishers, Inc.
Moreira, F. G., Silveira, D. X. D., & Andreoli, S. B. (2009). Knowledge and attitudes related to drug abuse and prevention displayed by public school educators. Revista Brasileira De Psiquiatria, 31(2), 95–100. https://doi.org/10.1590/s1516-44462009000200003
Nunoo, E. K. (2019). Sustainability Assessment in Ghana’s higher educational Institutions using the assessment questionnaire as a tool. In Encyclopedia of Sustainability in Higher Education (pp. 1584-1595). Cham: Springer International Publishing.
Richland, L. A., Frausel, R., & Begolli, K. N. (2016). Cognitive Development. The SAGE Encyclopedia of Theory in Psychology, 1–6. https://doi.org/10.4135/9781483346274.n50
Ruiz, J., & Snoeck, M. (2018). Adapting Kirkpatrick’s evaluation model to technology enhanced learning. Proceedings of the 21st ACM/IEEE International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings. https://doi.org/10.1145/3270112.3270114
Sarte, N. M. R., Santiago, B. T., Dagdag, J. D., & Handrianto, C. (2021). Welcome back: The return of college dropouts to school. Jurnal Pendidikan dan Pemberdayaan Masyarakat (JPPM), 8(2), 140-149. https://doi.org/10.36706/jppm.v8i2.15386
Schrader, P.G.,& Lawless, K.A. (2004). The knowledge, attitudes, & behaviors approach how to evaluate performance and learning in complex environments. Performance Improvement. 43(9),8-15. http://dx.doi.org/10.1002/pfi.4140430905
Schreiber, J. B., Nora, A., Stage, F. K., Barlow, E. A., & King, J. (2006). Reporting Structural Equation Modeling and Confirmatory Factor Analysis Results: A Review. The Journal of Educational Research, 99(6), 323–338. https://doi.org/10.3200/joer.99.6.323-338
Shek, D. T., & Yu, L. (2014). Confirmatory factor analysis using AMOS: A demonstration. International Journal on Disability and Human Development, 13(2), 191-204. https://doi.org/10.1515/ijdhd-2014-0305
Sicam, E. B., Umawid, M. D., Colot, J. D., Dagdag, J. D., & Handrianto, C. (2021). Phenomenology of parenting while schooling among filipino college student mothers in the province. Kolokium, 9(2), 80-94. https://doi.org/10.24036/kolokium-pls.v9i2.483
Stephens, P., Sloboda, Z., & Kenne, D. (2017). Evaluation of Substance Abuse Prevention and Treatment Programs. Research Methods in the Study of Substance Abuse, 411–440. https://doi.org/10.1007/978-3-319-55980-3_20
Tan, R. H. J., Yee, A., Sulaiman, A. H., Said, M. A., Danaee, M., & Lua, A. C. (2018). Effects of a school-based substance use prevention program on students in Malaysia. Journal of Health and Translational Medicine, 21(1). http://eprints.um.edu.my/20894/
Tripathi, J. P., & Artibansal. (2017). A Literature Review on Various Models for Evaluating Training Programs. IOSR Journal of Business and Management, 19(11), 14–22. https://doi.org/10.9790/487X-1911041422
Uto, M., & Ueno, M. (2020). A generalized many-facet Rasch model and its Bayesian estimation using Hamiltonian Monte Carlo. Behaviormetrika, 47(2), 469–496. https://doi.org/10.1007/s41237-020-00115-7
Valente, H., Martins, D., Pinto, M., Fernandes, L., & Barratt, M. J. (2022). A longitudinal study of behavioural outcomes following a visit to the Boom Festival 2018 drug checking service: individual and group level results. Drugs: Education, Prevention and Policy, 1-10.
Vinther, A. S. (2021). Harnessing the global expertise in drug use and drug prevention in physical activity settings: results from the Anabolic Steroid Prevention Survey. Drugs: Education, Prevention and Policy, 1-14.
Walid, D. A., Jusoh, A. J., & Omar, S. Z. (2021). A Review of Drug Prevention Program: Implementation in Several Countries. International Journal of Education, Information Technology and Others (IJEIT), 4(3), 500–511. https://doi.org/10.5281/zenodo.5446285
Watkins, M. W. (2018). Exploratory Factor Analysis: A Guide to Best Practice. Journal of Black Psychology, 44(3), 219–246. https://doi.org/10.1177/0095798418771807
Watson, J. C. (2017). Establishing Evidence for Internal Structure Using Exploratory Factor Analysis. Measurement and Evaluation in Counseling and Development, 50(4), 232–238. https://doi.org/10.1080/07481756.2017.1336931
Wulandari, I. A. I., Muhdhar, M. H. I. A., Mardiyanti, L., Triani, L., & Purwanti, W. C. (2021). The relationship between knowledge and attitude towards students’ behaviour in environmental literacy. The 4th International Conference on Mathematics and Science Education (ICoMSE) 2020: Innovative Research in Science and Mathematics Education in the Disruptive Era. https://doi.org/10.1063/5.0043110
Yong, A. G., & Pearce, S. (2013). A beginner’s guide to factor analysis: Focusing on exploratory factor analysis. Tutorials in Quantitative Methods for Psychology, 9(2), 79-94.
