Reliability, Sensitivity, and Specificity Measures of the Comprehensive Assessment of Accentedness and Intelligibility (CAAI) Test Battery
DOI:
https://doi.org/10.14738/assrj.115.17016Abstract
Purpose: This paper aims to introduce and report on measures of validation of a diagnostic assessment, the Comprehensive Assessment of Accentedness and Intelligibility (CAAI) Test Battery that identifies communication issues related to a person’s spoken accent. Due to scant availability of standardized assessments of accents, this test battery was developed to enable evidence-based diagnoses and determine severity levels of dialect- and accent-related communication difficulties (Shah, 2024). The present paper reports on psychometric testing conducted on the test battery with the resulting measures of reliability, sensitivity, and specificity of the CAAI. Methods: Native and nonnative speakers of English (N = 61) were administered the CAAI Test Battery, and their communication was evaluated and scored. Descriptive statistics were used to examine test sensitivity, specificity, interrater reliability, inter-item (internal consistency) reliability, and test-retest reliability. Results: The CAAI Test Battery meets high standards for test sensitivity in identifying accent-related communication concerns. Test specificity was high for not falsely identifying native speakers as having accent-related concerns. The instrument had strong interrater reliability across all 20 sections of the test (Pearson correlation coefficients ranged from 0.68 to 1.00, all highly significant at p<0.01). The test showed strong inter-item reliability/internal consistency with a large number of sections yielding moderate to high Cronbach’s alpha coefficient range of 0.70 to 1.00. The test showed strong test-retest reliability with correlation coefficients ranging from 0.73 to 1.00, significant at p<0.05. Conclusions: With high sensitivity, specificity, and reliability, the CAAI Test Battery is found to be a stable and meaningful means to assess dialect- and accent-related communication concerns. The CAAI Test Battery helps fill a crucial gap in the area of accents/dialects that lacks other validated assessment measures. With a sensitive and reliable assessment, clinicians and teachers can identify an accurate baseline pattern of errors to address for accent management or pronunciation teaching and achieve effective outcomes in a short amount of time.
References
American Speech-Language-Hearing Association. (1983, September). Social dialects and their implications [Position paper]. ASHA, 25, 23–27.
American Speech-Language-Hearing Association. (1985a, June). Clinical management of communicatively handicapped minority language populations [Position paper]. ASHA, 27, 29–32.
American Speech-Language-Hearing Association. (2007). Scope of practice in speech-language pathology. http://www.asha.org/policy/
Compton, A. (2002). Phonological assessment of foreign accent. San Francisco: Carousel House.
Flege, J. E., Munro, M. J., & MacKay, I. R. A. (1995). Effects of age of second-language learning on the production of English consonants. Speech Communication, 16 (1), 1-26.
Gu Y., & Shah A. (2019). A systematic review of interventions to address accent-related communication problems in healthcare. Ochsner Journal. 2019 Winter;19(4):378-396. doi: 10.31486/toj.19.0028
Maxwell, D. L., & Satake, E. (2006). Research and statistical methods in communication sciences and disorders. Clifton Park, NY: Thomson Delmar Learning.
Nunnally, J. & Bernstein, I. (1994). Psychometric theory. New York: McGraw-Hill.
Piske, T., MacKay, I. R. A., & Flege, J. E. (2001). Factors affecting degree of foreign accent in an L2: A review. Journal of Phonetics, 29, 191-215.
Rast, R., & Dommergues, J. Y. (2003). Towards a characterization of saliency on first exposure to a second language. In EUROSLA Yearbook (Vol. 3, pp. 131-156). Amsterdam: John Benjamins Publishing.
Schmidt, A. M., & Sullivan, S. (2003). Clinical training in foreign accent modification: A national survey. Contemporary Issues in Communication Science and Disorders, 30, 127-135.
Shah, A. P. (2005). Accent modification: Is it efficacy-based? Results from a nationwide survey. Paper presented at the Annual conference of the American Speech-Language-Hearing Association (ASHA), San Diego, CA, abstract in ASHA Leader, 10 (11), p. 86, 0513.
Shah, A. P. (2007a). Comprehensive Assessment of Accentedness and Intelligibility (CAAI). EBAM Institute, L.L.C. https://caaiassessment.com/
Shah, A. P. (2007b). A data-driven approach to comprehensive assessment of foreign-accented speech. Paper presented at the Annual conference of the American Speech-Language-Hearing Association (ASHA), Boston, MA, abstract in ASHA Leader, 12 (11), pp. 105, 0683.
Shah, A. P. (2009a). Adopting evidence-based practices in accent modification: A data-based assessment model. Speech Acoustics and Perception Laboratory Working Papers on Cross-Language Research, No. 5, 1-41.
Shah, A. P. (2009b). Validation study of the Comprehensive Assessment of Accentedness & Intelligibility (CAAI). Speech Acoustics and Perception Laboratory Working Papers on Cross-Language Research, No. 4, 1-23.
Shah, A. P. (2010). Comprehensive Assessment of Foreign-Accented Speech. A peer-reviewed, national web seminar produced by the Professional Development office of the American Speech & Hearing Association, Rockville, MD. ASHA.org. Web seminar offered as a Continuing Education course from 2011-2017.
Shah, A. P. (2023a). Dr. JN: An Adult Nonnative Speaker of English: High Proficiency. The Communication Disorders Casebook: Learning by Example, 411-420. Plural Publishing, Inc. San Diego, CA.
Shah, A. P. (2023b). Ms. PW: An Adult Nonnative Speaker of English: Low Proficiency. The Communication Disorders Casebook: Learning by Example, 421-443. Plural Publishing, Inc. San Diego, CA.
Shah, A. P. (2024). Assessing Accented Speech with a Data-Based Assessment Framework: The Key to Evidencebased Accent-Management. Advances in Social Sciences Research Journal, 11(5). 135-152.
Sikorski, L. (2002). Proficiency in oral English communication (POEC). Santa Ana, CA: LDS & Associates.
Sound Forge (1991-1998). Version 4.5 [Computer software]. Madison, WI: Sonic Foundry. http://www.sonicfoundry.com/.
Streiner, D. L. & Norman, G. R. (2003). Health Measurement Scales: A Practical Guide to Their Development and Use. New York: Oxford University Press.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Amee P. Shah
This work is licensed under a Creative Commons Attribution 4.0 International License.
