Indirect skill assessment using AI technology
DOI:
https://doi.org/10.14738/assrj.84.10077Abstract
In the new context of transitional, mobilised and globalised labour markets, an urgent need has emerged for meaningful assessment tools, methods and techniques to measure and recognize the workers’ skills. This paper aims to present a novel approach in skill assessment developed under NADINE H2020 project, the indirect skill assessment, as an alternative or supplement to the traditional selection methods. AI technology is utilised to facilitate the indirect skill assessment via two agnostic content serious games (Tetris and 2048), based on the evaluation of an individual’s performance in playing a game outside a situational judgement test (SJT) context. Novel datasets have been developed, comprised of game sessions and the corresponding skill assessments of the players through validated psychometric questionnaires, which were the basis for the algorithm training that would provide the estimation of a player’s skillset. The trained neural models for both games proved to have strong skill assessment capabilities, indicating that there is indeed a correlation between a person’s action sequence and his/her different skills.
References
Blázquez M. (2014). Skills-Based Profiling and Matching in PES. Luxembourg: Publications Office of the European Union.
Wayne, M. (2010). Human Resources Administration - 11th Edition. Mexico: Pearson Education.
Vargas, L. F. (2016). Construcción de pruebas psicométricas: aplicaciones a las ciencias sociales y de la salud Livia, J. & Ortiz, M. (2014) Lima: Editorial Universitaria. 166 p. Revista Digital de Investigaciónen Docencia Universitaria, 10(2), 92-94.
Chipana-Castillo, C., Miranda-Roca, GJ. & Vicente-Ramos, W. (2021). Effectiveness of psychometric tests for the selection of personnel in jobs in the retail sector, Management Science Letters 11, 1583–1590.
González, F. (2007). Psychological evaluation instruments. Havana: Medical Sciences.
Andrews, J. & Higson, H. (2008) Graduate Employability, ‘Soft Skills’ Versus ‘Hard’ Business Knowledge: A European Study, Higher Education in Europe, 33:4, 411-422 https://doi.org/10.1080/03797720802522627
Schulz, B. (2008). The importance of soft skills: Education beyond academic knowledge. Nawa Journal of Communication, 2(1), 146-154.
Schmitt N. (2014). Personality and cognitive ability as predictors of effective performance at work. Annual Review of Organizational Psychology and Organizational Behavior, 1(1), 45–65. https://doi.org/10.1146/annurevorgpsych-031413-091255
Ferro, R. & Vives, C. (2004). An analysis of the concepts of effectiveness, efficacy and efficiency in psychology, Tremédica, 5 (16). 97-99.
Dewson S., Eccles J., Tackey N.J., Jackson A., (2000). Guide to Measuring Soft Outcomes and Distance Travelled. UK: The Institute for Employment Studies, Department for Education and Employment, Crown. Available at http://www.employabilityinscotland.com/media/83581/guide-to-measuring-soft-outcomes-distancetravelled.pdf
Morgeson, F., Campion, M., Dipboye, R., Hollenbeck, J., Murhpy, K., & Schmitt, N. (2007). Are we getting fooled again? coming to terms with limitations in the use of personality tests for personnel selection, Personnel Psychology, 60, 1029-1049. https://doi.org/10.1111/j.1744-6570.2007.00100.x
Nikolaou, I., Georgiou, K. & Kotsasarlidou, V. (2019). Exploring the Relationship of a Gamified Assessment with Performance, The Spanish Journal of Psychology, 22. https://doi.org/10.1017/sjp.2019.5.
Hausknecht J., Day D. & Thomas S. (2004). Applicant Reactions to Selection Procedures: An Updated Model and Meta‐Analysis, Personnel Psychology, 57. https://doi.org/10.1111/j.1744-6570.2004.00003
Sinickas, A. (2007). Finding a cure for survey fatigue, Strategic Communication Management, 11(2), 11.
Weekley, J.A. and Ployhart, R.E. (2006). Situational Judgment Tests: Theory, Measurement and Application, Jossey-Bass, San Francisco, CA.
Herde, C. N., Lievens, F., Solberg, E. G., Harbaugh, J. L., Strong, M. H., & Burkholder, G. J. (2019). Situational judgment tests as measures of 21st century skills: Evidence across Europe and Latin America. Journal of Work and Organizational Psychology, 35(2), 65–74.
Weekley, J.A. and Jones, C. (1997). Video-based situational testing, Personnel Psychology, Vol. 50, No. 1, pp. 25-49. URL: http://dx.doi.org/10.14738/assrj.84.10077 735
Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37(4), 426–441. https://doi.org/10.1108/00483480810877598
Heine, S. J. & Buchtel, E. E. (2009). Personality: The universal and the culturally specific. Annual Review of Psychology, 60, 369-394. https://doi.org/10.1146/annurev.psych.60.110707.163655
Lytle, A. L., Brett, J. M., Barsness, Z. I., Tinsley, C. H., & Janssens, M. (1995). A paradigm for confirmatory crosscultural research in organizational behavior. Research in Organizational Behavior, 17, 167-214.
Djaouti, D., Alvarez, J. & Jessel, J.P. (2011). Classifying Serious Games: the G/P/S model, Handbook of Research on Improving Learning and Motivation through Educational Games: Multidisciplinary Approaches. IGI Global.
Lugmayr, A., Suhonen, J., Hlavacs, H., Montero, C., Suutinen, E. & Sedano, C. (2016). "Serious storytelling - a first definition and review". Multimedia Tools and Applications. 76 (14): 15707–15733. https://doi.org/10.1007/s11042-016-3865-5.
Bhatia, S. (2018). Applicant Reactions to Game-Based Assessments: The Effects of Flow, Fairness, and Fit, MI: Michigan State University, Lansing.
Ellison, L.J., McClure Johnson, T., Tomczak, D., Siemsen, A. and Gonzalez, M.F. (2020), "Game on! Exploring reactions to game-based selection assessments", Journal of Managerial Psychology, Vol. 35 No. 4, pp. 241-254. https://doi.org/10.1108/JMP-09-2018-0414
Glover, I. (2013). Play as you learn: Gamification as a technique for motivating learners. In J. Herrington et al. (Eds.), Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications (pp. 1999-2008). VA: AACE.
DuVernet, A. M., &Popp, E. (2014). Gamification of workplace practices. The Industrial-Organizational Psychologist, 52, 39–44.
Van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless. Educause Review, 41, 16. Retrieved from https://www.researchgate.net/
de Klerk, S., Veldkamp, B. P., & Eggen, T. J. (2015). Psychometric analysis of the performance data of simulationbased assessment: A systematic review and a Bayesian network example, Computers & Education, 85, 23–34. https://doi.org/10.1016/j.compedu.2014.12.020
Bellotti, F., Kapralos, B., Lee, K., Pablo Moreno-Ger, &Ber-ta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Inter-action, 2013 https://doi.org/10.1155/2013/136864.
Mislevy, R. J., Corrigan, S., Oranje, A., DiCerbo, K., Bauer, M. I., von Davier, A., Drasgow, F. (2016). Psychometrics and game-based assessment. In F. Drasgow (Eds.), Technolo-gy and Testing: Improving Educational and Psychological Measurement (pp. 23–48). New York, NY: Routledge.
Chamorro-Premuzic T., Akhtar R., Winsborough D., & Sherman R. A. (2017). The datafication of talent: How technology is advancing the science of human potential at work. Current Opinion in Behavioral Sciences, 18, 13–16. https://doi.org/10.1016/j.cobeha.2017.04.007.
Kato & de Klerk, 2017) Kato P. M. & de Klerk S., (2017) Serious Games for Assessment: Welcome to the Jungle, Journal of Applied Testing Technology, Vol 18(S1), 1-6, 2017.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests, Psychometrika, 16(3), 297–334. https://doi.org/ 10.1007/BF02310555
Schell, J. (2008). The art of game design. Burlington, MA: Morgan Kaufman.
Baron, T., Heath, C. & Amresh, A. (2016). Towards a context agnostic platform for design and assessment of educational games. In Connoly T. & Boyle D. (Eds.), Proceedings of the European Conference on Games Based Learning (pp. 34-40). Scotland, GB: The University of West Scotland. th
Tremblay, J. & Verbrugge, C. (Eds.). (2015). A game genre agnostic framework for game-design. IEEE/ACM 4 International Workshop on Games and Software Engineering, Florence, Italy. https://doi.org/ 10.1109/GAS.2015.9
Kankaraš, M. (2017). Personality matters: Relevance and assessment of personality characteristics, OECD Education Working Papers, No. 157, OECD Publishing, Paris. http://dx.doi.org/10.1787/8a294376-en
Stiegler, A., & Zimmermann, G. (2015). Gamification and accessibility. In J. Zhou and G. Salvendy (Eds.), Human Aspects of IT for the Aged Population: Design for Aging (pp. 145-154). https://doi.org/ 10.1007/978-3-319-20892-3_15
Neller, T. (2015). Pedagogical possibilities for the 2048 puzzle game. The Journal of Computing Sciences in Colleges, 30, 38-46. Retrieved from https://cupola.gettysburg.edu/
Bannon, S., Ford, K., & Meltzer, L. (2011). Understanding millennials in the workplace. The CPA Journal, 81(11), 61–65. Retrieved from: https://www.nysscpa.org/ news/publications/the-cpa-journal/articlepreview?ArticleID=10721
Gibson, L. A. & Sodeman, W. A. (2014). Millennials and technology: Addressing the communication gap in education and practice. Organization Development Journal, 32(4), 63–75.
Gkorezis P., Georgiou K., Nikolaou I. & Kyriazati A. (2020): Gamified or traditional situational judgement test? A moderated mediation model of recommendation intentions via organizational attractiveness, European Journal of Work and Organizational Psychology, https://doi.org/ 10.1080/1359432X.2020.1746827
Cutumisu, M., Blair, K., Chin, D., & Schwartz, D. (2015). Posterlet: A game-based assessment of children’s choices to seek feedback and to revise. Journal of Learning Analytics, 2, 49-71. https://doi.org/ 10.7939/R31R6ND7N
Baron, T. (2017). An architecture for designing content agnostic game mechanics for educational burst games (Doctoral dissertation, Arizona State University).
Mislevy, R., Steinberg, L., & Almond, R. (2003). Focus article: On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3-62. https://doi.org/10.1207/S15366359MEA0101_02
Tippins, N. T., Oswald, F. L., & McPhail, S. M. (2021). Scientific, Legal, and Ethical Concerns about AI-Based Personnel Selection Tools: A Call to Action, PsyArXiv, 28. https://doi.org/10.31234/osf.io/6gczw
Armstrong, M. B., Landers, R. N., & Collmus, A. B. (2015). Gamifying recruitment, selection, training, and performance management: Game-thinking in human resource management. In Emerging Research and Trends in Gamification (pp. 140-165). IGI Global. https://doi.org/10.4018/978-1-4666-8651-9.ch007.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2021 MARIA TOUNTOPOULOU
This work is licensed under a Creative Commons Attribution 4.0 International License.
