Validation of Mentee-Teachers’ Assessment Tool within the Framework of Generalisability Theory at the Faculty of Education, University for Development Studies
DOI:
https://doi.org/10.14738/assrj.107.14968Keywords:
Object of measurement, Relative decision, Absolute decision, universe of generalisation, universe of admissible observations, Composite facet, Generalisability study, Decision(D) Study, OptimisationAbstract
pPractitioners in assessment and other researchers have over the years expressed dissatisfaction with the lack of consistency in scores obtained from use of multiple measurement instruments. Such scores derived from these largely inconsistent and unreliable procedures are relied upon by decision makers in taking very important decisions in education, health and other related fields. The purpose of this study therefore, was to apply the G theory procedures to validate the mentee assessment tool being used at the Faculty of Education, University for Development Studies. The G study involved estimating the generalisability (reliability-like) coefficients of mentees’ assessment scores, and determining the level of acceptability (validity) of these coefficients. A nested design was used because different sets of raters assessed different student-mentees on different occasions in the field. The relationship among these variables (facets); students, raters and occasions, appropriately mirrored a nested relationship. Data obtained by raters on 300 students, in the 2018/2019 off-campus teaching practice, were entered into EDUG software for relevant analysis of the results. The study found that both rater and student facets accounted for the largest measurement errors in mentees’ observed scores, reporting an estimated G coefficient of 0.62(62%), and representing a positive moderate relationship. Based on these findings, the study concluded that, the quality of mentee observed scores could be improved for either relative or absolute decisions by varying the number of levels of both raters and occasions. To achieve acceptable G coefficient values of 0.83 and above, it is recommended that, decision makers employ a model that uses four raters per occasion for three occasions of assessment./p pstrongem /em/strong/pReferences
Adom, D., Husein, E. K., & Agyem, J. A. (2018). Theoretical and conceptual framework: Mandatory ingredients of a quality research. International journal of scientific research. https://www.researchgate.net/publication/322204153
Alexopoulos, D.S. (2007). Classical test theory. In N.J. Salkind (Ed.), Encyclopedia of measurement and statistics. Thousand oaks: CA: Sage publications, 140-143.
Allal, L., & Cardinet, J. (1997). Generalisability theory. In J.P. Keres (Ed.), Educational research, methodology, and measurement: An international handbook (2 , p. 737 -741). Cambridge, United Kingdom: Cambridge University.
Allen, M. J. & Yen, W. M. (2002). Introduction to measurement theory. Illinois: Waveland press, Inc.
Ali, A. M., & Yusof, H. (2011). Quality in qualitative studies: the case of validity, reliability and generalisability. Issues in social and environmental accounting, 5(1), p.25 -64.
Alkharusi, H. (2012). Generalisability theory: An analysis of variance approach to measurement problems in educational assessment. Journal of studies in education. https://www.researchgate.net/publication/256349585
Ary, D., Jacobs, L. C., Sorensen, C. K., & Walker, D. A. (2014). Introduction to research in education. Wadsworth, Cengage Learning Inc.
Atilgan, H. (2019). Reliability of essay ratings: A study on generalisability theory. Eurasian journal of educational research, 1, 133 – 150.
Atilgan, H. (2008). Using generalisability theory to assess the score reliability of the special ability selection examinations for music education programmes in higher education. International Journal of research &methods in education, 31 (1), 63 –76 https://doi.org/10.1080/17437270801919925.
Baartman, L. K.J., Bastiaenns, T. J., Kirschner, P. A., van der Vleuten, C. P. M. (2007). Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks. Educational research review; www.elsevier.com/locate/EDUREV
Babie, E. (2013). The practice of social research (13 ed.). Canada: Wadsworth, Cengage Learning.
Baker, F. B. (2001). The basics of item response theory (2 ed.). Washington, DC. Eric publications.
Bluman, A. G. (2012). Elementary statistics: A step-by-step approach (8 ed.). USA, McGraw-Hill.
Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model; Fundamental measurement in the human sciences (3 ed.) New York, Routeledge group.
Bouwer, R., Beguin, A., Sanders, T., & van den Berg, H. (2015). Effect of genre on the generalisability of writing scores. Language testing, 32 (1), 83 – 100.
Brannan, R. L. (2013). Commentary on ‘validating the interpretations and uses of test scores. Journal of educational measurement, 50: 74-83.
Brennan, R. L. (2011). Generalisability theory and classical test theory. Centre for advanced studies in measurement and assessment, University of Iowa. Applied measurement in education,24: 1-21.
Brennan, R.L. (1996). Generalisability of performance assessments. In technical issues in large-scale performance assessment, edited by Philips, G., 19 -58. Washington, DC: National center for educational statistics.
Brennan, R. L. (2003). Coefficients and indices in Generalisability theory. Center for advanced studies in measurement and assessment (CASMA) research report (1).
Brennan, R.L. (2001). Generalisability theory. New York: Springler-Verlag New York, Inc.
Brennan, R. L. (1998). A perspective on the history of generalizability theory. Educational measurement: issues and practice,16(4),14-20
Brennan, R.L. (1996). Generalisability of performance assessments. Technical issues in large-scale performance assessment. Washington, DC: National center for education and statistics.
Brennan, R.L. (1992). Elements of generalisability (2 ed.) Iowa city. ACT publications.
Iddrisu, S. A. (2023). Validation of Mentee-Teachers’ Assessment Tool within the Framework of Generalisability Theory at the Faculty of Education,
Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalisability theory using Edu G. Quantitative methodology series. New York, Routledge Group.
Cook, D.A., Thompson, W.G., & Thomas, K.G. (2014). Test-enhanced web-based learning: optimizing the number of questions (a randomized crossover trial). Academic med: 169 -175
Cook, D.A., Beckman, T. (2006). Current concepts in validity and reliability for psychometric instruments: theory and application. Journal of education, 19(7), 166.
Crocker, L., & Algina, J. (2008). Introduction to classical and modern test theory. OH, Cengage learning.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth group/Thomson learning.
Cronbach, J.L. (1984). Essentials of psychological testing. New York: Happers & Row publishers.
Cronbach, J. L., Lin, R. L., Brenan, R.L. & Haertel, E. (1997). Generalisability analysis for performance assessment of student achievement or school effectiveness. Educational and psychological measurement, 57, 373 -399.
Denegar, C. R., & Ball, D. W. (1993). Assessing reliability and precision of measurement: an introduction to intra class correlation and standard error of measurement. Journal sport rehabil, 2 (1),35- 42.
DeVillis, R.F. (2017). Classical test theory. Med care, 44(11), 50 – 59.
Dogan, C.D., & Uluman, M. (2017). A comparison of rubrics and graded category rating scales with various methods regarding raters’ reliability. Educational sciences: Theory & practice,7, 631 -651. https: dx.doi.org /10.12738/estp.2017.2.0321.
Doowning, S.M. (2003). Validity on the meaningful interpretation of assessment data. Med education, 37: 830-837.
Doowning, S.M. & Haladya, T.M. (2004). Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ, 38: 327 -333.
DeShon, R. P. (2002). Generalisability theory. In F. Drasgow & N. Schmitt (Eds.), Measuring and analysing behaviour in organisations, 189 -220.
Edenborough, R. (1999). Using psychometrics, a practical guide to testing and assessment (2 ed.). London, Kogan page Ltd.
Feldt, L. S., & Brennan, R. L. (1989). Reliability. In R.L. Linn (Ed.), Educational measurement (3 ed., pp. 127 -144). New York: Macmillan.
Fleming, M., House, S., Hanson, V. S., Yu, L., Garbutt, J., McGee, R., Kroenke, K., Abedin, Z., & Rubio, D. M (2013). The mentoring competency assessment: Validation of a new instrument to evaluate skills of research mentors. Academic medicine, 88(7).
Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2015). How to design and evaluate research in education (9 ed.). New York, McGraw-Hill education.
Fulton, S. & Krainovich-Miller, B. (2010). Gathering and appraising the literature in LoBiondo-Wood, G. & Haber, J. (Eds). Nursing research: methods and critical appraisal for evidence-based practice (7th ed.). St, Luis MO: Mosby Elsevier.
Gaur, A. & Gaur, S.S. (2009). Statistical methods for practice in research: A guide to data analysis using SPSS. New Delhi, Sage publications Inc. URL: http://dx.doi.org/10.14738/assrj.107.14968 123
Gebril, A. (2009). Score generalisability of academic writing tasks: does one test method fit it all? Language testing, 26(4):507 – 531.
Ghana Education Service Headteacher’s handbook. Ghazali, N. H. (2016). A reliability and validity of an instrument to evaluate the school-based assessment system: A pilot study. International journal of evaluation and research in education (IJERE),5 (2), 148 – 157.
Gierl, M. J, & Bisanz, J. (2001). Item response theory for psychologists -Book review. Applied psychological measurement, 25 (4), 405 -408.
Graham, S., Harris, K., & Herbert, M. (2011). Informing writing: The benefits of formative assessment. A carnegie corporation time to act report. Washington, DC: Alliance for excellent education.
Grant, C. & Osandoo, A. (2014). Understanding, selecting and integrating a theoretical framework in dissertation research. Creating the blueprint for ‘House’. Administrative issues journal: connecting education, practice and research. pp.12 -22.
Gullo, D. F. (2005). Understanding assessment and evaluation in early childhood education (2 ed.). New York, Teachers college press.
Gugiu, P.C. (2011). Summative confidence. Unpublished doctoral dissertation, Western Michigan university, Kalamazoo.
Gugiu, M.R., Gugiu, P. C., & Baldus, R. M. A. (2012). Utilizing generalizability theory to investigate the reliability of the grades assigned to undergraduate research papers. Journal of multidisciplinary evaluation, 8(19),26-40.
Hafner, J. C., & Hafner, P. M. (2003). Quantitative analysis of the rubric as an assessment tool: An empirical study of student peer-group rating. International journal of science education, 25 (12), 1509 -1528.
Hambleton, R.K. & Jones, R.W. (1997). Comparison of classical test theory and item response theory and their applications to test development.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury; Sage publications.
Hambleton, R.K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development.
Hamrick, L.R., Haney, A. B., Kelleher, B. L., & Lane, S. P. (2020). Using generalisability theory to evaluate the comparative reliability of developmental measures in neurogenic syndrome and low-risk populations. Journal of neurodevelopmental disorders, 12 (16).
Heitman, R. J., Kovaleski, J.E., & Pugh, S. F. (2009). Application of generalizability theory in estimating he reliability of ankle-complex laxity measurement. National Athletic trainers’ Association, Inc. Journal of Athletic training, 44(1), 48 -52.
Hopkins, K. D. (1998). Educational and psychological measurement and evaluation (8 ed.). Boston, MA: Allyn and Bacon.
Howell, R. (2014). Grading rubrics: Hoopla or help? Innovations in education and teaching international. 51(4); 400 -410.
Ilgen, J.S., Hatala, R., & Cook, D.A. (2015). A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med education, 161 -173.
Iddrisu, S. A. (2023). Validation of Mentee-Teachers’ Assessment Tool within the Framework of Generalisability Theory at the Faculty of Education, Institute for Continuing Education and Interdisciplinary Research (ICEIR) (2012). University for Development Studies mentoring policy. Tamale, M-bukConcepts publication.
Jonsson, A. & Svingby, G. (2007). The use of scoring rubrics; reliability, validity and educational consequences. Educational research review 2. 130 – 144.
Kan, A. (2007). Effects of using a scoring guide on essay scores: Generalisability theory. Perceptual and motor skills, 105, 891 -905.https://doi.org/10.2466/pms.105.3.891 -905.
Kane, M.T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4 ed., pp. 17 -64). Westport, CT: American Council on Education /Praeger.
Kane, M.T. (2001). Current concerns in validity theory. Journal of education measurement, 38: 319 -342.
Kane, M.T. (1992). An argument-based approach to validity. Psychological bulletin, 112: 527 – 535.
Kaplan, R, M. & Saccuzzo, D.P. (1989). Psychological testing: Principles, applications and issues (2 ed.). California, Brooks/Cole publishing Co.
Kovaleski, J.E, Hollis, J. M., Heitman, R. J., Gurchiek, L. R., Pearsall, A. W, (IV). Assessment of ankle-subtalar-joint complex laxity using an instrumented ankle arthrometer: an experimental cadaveric investigation. Journal of athl train, 37(4), 467 -474. Kean, J. & Reilly, J. (2014). Classical test theory.192-194
Kim, Y. S. G., Schatschneider, C., Wanzek, J., Gatlin, B., Otaiba, S. (2017). Writing evaluation: rater and task effects on the reliability of writing scores for children in Grades 3 and 4. Read writ, 30, 1287- 1310.
Kline, J. B. (2014). Classical test theory: Assumptions, equations, limitations, and item analyses. Thousand Oaks, Sage publications.
Knight, P. (2006). The local practices of assessment; Assessment and evaluation in higher education.31(4);435 -452.
Kolen, M. J., & Brennan, R. L. (1995). Test equating methods and practices. New York, Springer – Verlag Inc.
Lawson, J.E. & Cruz, R. A (2017). "Evaluating special educators’ classroom performance: Does rater “type” matter?", assessment for effective intervention.
Leedy, P. O., & Ormrod, J. E. (2010). Practical research, planning and design (9 ed.). New Jersey, Pearson education Inc.
Lissitz, R. W., & Samuelsen, K. (2007). A suggested change in terminology and emphasis regarding validity and education. Educational researcher, 36, 437 - 448.
Mafla, A. C., Herrera-Lopez, H.M., & Villalobos-Galvis, F. H. (2018). Psychometric approach of the revised illness perception questionnaire for oral health (IPQ-R-OH) in patients with periodontal disease. Journal of periodontology. DOI: 10.1002/JPER.18-0136.
Magis, D. (2014). Accuracy of asymptotic standard errors of the maximum and weighted likelihood estimators of proficiency levels with short tests. Applied psychological measurement, 38, 105 -121.
Marcoulides, G., & Ing, M. (2013). The use of generalisability theory in language assessment. In A. Kunnan, (Ed.), The companion language assessment, 3, 1207 -1223. New York, NY: John Wiley & Sons, Inc. DOI: 10.1002/9781118411360.wbcla014
Marcoulides, G. A (1999). Generalisability theory: Picking up where the Rash IRT models leaves off? In.
Embretson, S & Hershberger, S. The new rules of measurement: What every psychologist and educator should know. Lawrence Eribaum associates Inc., 129 -130. Marcoulides, G. A. (1996) "Estimating variance components in generalizability theory: The covariance structure analysis approach", structural equation modeling: A multidisciplinary journal.
Marcoulides, G. A. (1993). Maximizing power in generalizability studies under budget constraints, Journal of educational statistics, 18(2), 197-206.
McDonald, R. P. (1999). Test theory. A unified treatment Mahwah, NJ: Lawrence Erlbaum Associates.
Medvedev, O.N., Merry, A.F., Skilton, C., Gargiulo, D. A., Mitchell, S. J., & Weller, J. M. (2019). Examining reliability of WHOBARS: a tool to measure the quality of administration of WHO surgical safety checklist using generalisability theory with surgical teams from three New Zealand hospitals. BMJ Open 2019;9:e022625:doi:10.1136/bmjopen - 2018-022625.
Messick, S. (1995). Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychological Association; 50:741 – 749.
Messick, S. (1989). Validity in educational measurement (3 ed). New York; American Council on education and Macmillan.
Moskal, B., & Leydens, J. (2000). Scoring rubric development: Validity and reliability. Practical assessment, research and evaluation,7(10),1-11.
Muijs, D. (2011). Doing quantitative research in education with SPSS. London: Sage Publications limited.
Naizer, G.(2007). Basic concepts in generalisability theory: a more powerful approach to evaluating reliability. http://www.eric.ed.gov. Eric document reproduction service no. ED341729. Accessed
Nitko, A. J., & Brookert, S. M. (2011). Educational assessment of student. Boston, MA: Pearson Education.
Nitko, A. J. (2004). Educational assessment of students (4 ed.). New Jersey, Pearson Education, Inc.
Nunnally, J., & Bernstein, I. (1994). Psychometric theory (3 ed.). New York: McGraw-Hill, Inc.
Ojerinde, D., Popoola, K., Oje, F., & Onyeneho, P. (2012). Introduction to item response theory (2 ed.). Abuja. Marvelouse mike press Ltd.
Orentz, J.O. (2018). Generalisability theory: A comprehensive method for assessing and improving the dependability of marketing measures. Journal of marketing research, 24, 19 -28.
Pallant, J. (2007). Step-by-step guide to data analysis using version 15: SPSS survival manual (3 ). McGraw-Hill, open university press.
Regan, B.G., & Kang, M. (2005). Reliability: issues and concerns. AthlTher today, 10(6), 30 -33. individual differences, 103, 32-39. Doi: 10.1016/j.paid.2016.04.007.
Revelle, W. (2013). Personality project. Retrieved October, 2013, from http://personalityproject.org/revelle/syllabi/405.old.syllabus.html
Sadler, D. (2009). “Indeterminacy in the use of preset criteria for assessment and grading”. Assessment and evaluation in higher education, 34 (2): 159 -165.
Sadler, P. M. & Good, E. (2006). The impact of self –and peer-grading on student learning. Educational assessment,11(1), 1 -31.
Shavelson, R. J. & Webb, N.M. (2005). Generalisability theory. British journal of mathematical and statistical psychology, 599- 612.
Shavelson, R.J, Webb, N., & Rowley, G. (1992). Generalisability theory. https: //www.researchgate.net/publication/344083148
Shavelson, R.J., & Webb, N.M. (1991). Generalizability theory: A primer. Newbury Park, Sage publications.
Sinclair, M. (2007). “A guide to understanding theoretical and conceptual frameworks.” Evidence-Based midwifery, 5 (2), 39. Gale onefile: Health and medicine. link.gale.com/apps/doc/A167108906/HRCA?u=anon~a99d 7734&sid=googleScholar&xid=20965ff (Accessed 5 May 2022).
Sireci, S. (2007). On validity theory and test validation. Educational researcher, 36(8), 477 -481.
Sireci, S. G., & Parker, P. (2006). Validity on trial: Psychometric and legal conceptualization of validity. Educational measurement: Issues and practice, 25 (3), 27 -34.
Stangor, C. (2006). Research methods for the behavioural sciences (3 ed). Boston: Houghton Mifflin.
Stegner, A.J., Tobar, D. A., & Kane, M. T. (1999). Generalizability of change scores on the body awareness scale. Measurement in physical education and exercise science.
Stemler, S. E. (2007). Cohen’s kappa. In N.J. Salkind (Ed.), Encyclopedia of measurement and statistics. Thousand Oaks, CA: Sage publications.
Steyer, R. (1999). Classical (Psychometric) test theory. Jena, Germany.
Steyer, R., Schmitt, M., & Eid, M. (1999). Latent state–trait theory and research in personality and individual differences. European journal of Personality,13(5),389 –408.
Sudweeks, R., Reeve, S. & Bradshaw, W. (2005). A comparison of G theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing writing Vol. 9, 239-261.
Suen, H.K. & Lei, P. W. (2007). Classical versus generalizability theory of measurement. Journal of Educational measurement,4, 1-13.
Schoonen, R. (2012). The validity and generalisability of writing scores. The effect of rater, task and language. In E. Van Steendam, M. Tillema, G. Rijlaarsdam, & H. van den Berg (Ed.), Measuring writing: Recent insights into theory, methodology and practice (pp. 1- 22). Leiden, The Netherlands: Brill.
Stevens, D. D., & Levi, A. J. (2005). Introduction to rubrics. Sterling, VA: Stylus publishing.
Strube, M.J. (2002). Reliability and generalisability theory. In L.G grimme & P.R. Yarnold (Eds.) Reading and understanding more multivariate statistics (pp. 23 -66). Washington, DC: American Psychological Association.
Swiss Society for Research in Education Working Group (2010). Quality of measurement in education- EDUG user guide. Switzerland.
Tabachnick, B.G., & Fidell, L.S. (2007). Using multivariate statistics (5 ed.). Allyn and Bacon/Pearson education.
Tan, K. & Prosser, M. (2004). Qualitatively different ways of differentiating student achievement: A phenomenographic study of academic’s conceptions of grade descriptors. Assessment and evaluation in higher education, 29(3), 267 -282. URL: http://dx.doi.org/10.14738/assrj.107.14968 127
Tavares, W., Bridges, R. & Tuner, L. (2018). Applying Kane’s validity framework to a simulation- based assessment of clinical competence. Advances in health science education. https//www.researcgate, net /publication/320679278
Templin, J. (2012). Item response theory. Colorado, Georgia, USA.
Thompson, B. (2003). A brief introduction to generalisability theory. In B. Thompson (Eds.) Score reliability: Contemporary thinking on reliability issues, 43 -58. Thousand Oaks, C.A: Sage.
Turgut, M. & Baykul, Y. (2010). Measurement and evaluation in education (Egitimde olome ve degerlendirme). Ankara: Pegem Akademi. University for Development Studies Administrative Manual, (2018). University for
Development Studies Statutes, (2017).
Vispoel, W. P., Moris, C. A., & Kilinc, M. (2018). Applications of generalisability theory and their relations to classical test theory and structural equation modeling. American psychological association; Psychological methods, 23(1),1-26.
Webb, N.& Shavelson, R.J. (2005). Generalisability theory: Overview. https://www.researchgate,net/publication/22758011
Webb, N.M., Rowley, G.L., & Shavelson, R.J. (1988). Sing generalisability theory in counselling and development. Measurement and evaluation in counselling and development, 21, 81-90.
Yukawa, M., Gansky, S.A., O’Sullivan, P., Teherani, A., & Fieldman, M.D. (2020). A new mentor evaluation tool: Evidence of validity. Plos one 15 (6): e0234345. https://doi.org/10.1371/journal.pone.0234345. Accessed on June 16,2020.
Zanon, C., Claudio, S. H., Hanwook, Y., & Hambleton, R. K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexaoe critica.29:18. DOI 10,186/54:1155 – 016 -0040 -x.
Zhang, B., Johnston, L., & Kilinc, G. (2008). Assessing the reliability of self-and peer rating in student group work. Assessing and evaluation in higher education,33(3), 329– 340.
Zumbo, B. D., Gadermann, A. M., & Zeisser, C. (2007). Ordinal versions of coefficients alpha and theta for Likert rating scales. Journal of modern applied statistical methods, 6(1), 21 -29.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Simon Alhassan Iddrisu
This work is licensed under a Creative Commons Attribution 4.0 International License.
