The single-task writing assessments used in longitudinal CDST studies sketching L2 writing development have raised concerns regarding their reliability. By means of Generalizability Theory (GT), a statistical theory evaluating the reliability of behavioural assessments (Brennen, 2001; Shavelson & Webb, 2005), this study investigated a) the reliability of computer-assessed single-task writing assessments and b) the effects of task topics and task-taking occasions on assessment reliability. In this study, we compared six types of scores obtained from different language traits, through a 1-day dataset and a 21-day dataset consisting of five IELTS writing tasks. In total, 90 essays from 18 Chinese learners of English were rated on four CAF measures and two cohesion measures. The scores of such measures were analysed with GT using R 3.6.1 (R Core Team, 2019) with the gtheory package (Moore, 2016) and EXCEL. The results show that single-task assessments are reliable when they are scored on some language traits (e.g. accuracy and fluency), while assessments containing multiple tasks do not assure reliability if scored on traits such as cohesion. The reliability of scores on all traits can be improved by increasing the number of tasks in an assessment, but not always up to a desired level. The impact of task topics and task-taking occasions on the reliability of scores, on the other side, significantly differ from one trait to another. These results direct future CDST research to the difference in the reliability of independent language traits resulted from the dynamics of a language system, rather than the reliability of writing assessments by and large. Such studies could assist in selecting traits to be rated and interpreting the score variance between assessments when measuring the development of writing skills.
|Publication status||Accepted/In press - 20-Mar-2021|
|Event||American Association for Applied Linguistics - Vitual Conference|
Duration: 20-Mar-2021 → 23-Mar-2021
|Conference||American Association for Applied Linguistics|
|Period||20/03/2021 → 23/03/2021|