Reliability, Sensitivity, and Specificity Measures of the Comprehensive Assessment of Accentedness and Intelligibility (CAAI) Test Battery

Abstract

Purpose: This paper aims to introduce and report on measures of validation of a diagnostic assessment, the Comprehensive Assessment of Accentedness and Intelligibility (CAAI) Test Battery that identifies communication issues related to a person’s spoken accent. Due to scant availability of standardized assessments of accents, this test battery was developed to enable evidence-based diagnoses and determine severity levels of dialect- and accent-related communication difficulties (Shah, 2024). The present paper reports on psychometric testing conducted on the test battery with the resulting measures of reliability, sensitivity, and specificity of the CAAI. Methods: Native and nonnative speakers of English (N = 61) were administered the CAAI Test Battery, and their communication was evaluated and scored. Descriptive statistics were used to examine test sensitivity, specificity, interrater reliability, inter-item (internal consistency) reliability, and test-retest reliability. Results: The CAAI Test Battery meets high standards for test sensitivity in identifying accent-related communication concerns. Test specificity was high for not falsely identifying native speakers as having accent-related concerns. The instrument had strong interrater reliability across all 20 sections of the test (Pearson correlation coefficients ranged from 0.68 to 1.00, all highly significant at p<0.01). The test showed strong inter-item reliability/internal consistency with a large number of sections yielding moderate to high Cronbach’s alpha coefficient range of 0.70 to 1.00. The test showed strong test-retest reliability with correlation coefficients ranging from 0.73 to 1.00, significant at p<0.05. Conclusions: With high sensitivity, specificity, and reliability, the CAAI Test Battery is found to be a stable and meaningful means to assess dialect- and accent-related communication concerns. The CAAI Test Battery helps fill a crucial gap in the area of accents/dialects that lacks other validated assessment measures. With a sensitive and reliable assessment, clinicians and teachers can identify an accurate baseline pattern of errors to address for accent management or pronunciation teaching and achieve effective outcomes in a short amount of time.