The CEFR Rating Scale Functioning: An Empirical Study on Self- and Peer Assessments
DOI:
https://doi.org/10.11113/sh.v9n4-2.1355Keywords:
Common European Framework of Reference (CEFR), self-assessment, peer assessment, rating scale functioningAbstract
One of the criticisms on the Common European Framework of Reference (CEFR) rating scales pertains to the lack of reference to the performance of learners in the construction process of the scales. Therefore, this study attempted to delve into rating scale functioning by English as a Second Language (ESL) learners during self-assessment and peer assessment of their oral proficiency practice. Two objectives guided the study: 1) to gauge the overall rating scale functioning and 2) to measure each criterion scaling structure. Three self- and peer assessments’ cycles were conducted in three months. In each cycle, eleven learners recorded their own speech, uploaded their video clips to a private YouTube channel and assessed their own videos as well as selected peers based on five CEFR oral assessment criteria with six levels of ratings (A1-C2). Findings revealed that four of the CEFR levels were utilised (B1-C2). Categories A1 and A2 (basic user level) however, were not observed during the practice. Analysis from the Many-Facet Rasch Measurement (MFRM) indicated that utilised categories seemed to function usefully since each category observed was advancing by more than 1.4 logits. Category B2 dominated four criteria of ratings awarded while B1 dominated the rating distribution for fluency. The implications of this study will be discussed in relation to rating scale development, specifically on matching learners’ proficiency to the psychometrically developed rating criteria as well as illustrating assessment as learning approach in the ESL classroom where learners become the key assessors for their own performance.Â
References
Alderson, J. C., Figueras, N., & Kuijper, H. (2006). Analysing Tests of Reading and Listening in Relation to the Common European Framework of Reference : The Experience of The Dutch CEFR Construct Project. Language Assessment Quarterly, 3(1), 3–30.
Bosker, H. R., Pinget, A.F., Quene, H., Sanders, T., & de Jong, N. H. (2012). What makes Speech Sound Fluent? The Contributions Of Pauses, Speed And Repairs. Language Testing, 30(2), 159–175.
Brantmeier, C., Vanderplank, R., & Strube, M. (2012). What About Me? System, 40(1), 144–160.
Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press.
Davis, L. (2015). The Influence Of Training And Experience On Rater Performance In Scoring Spoken Language. Language Testing. 33(1), 117-135
Deygers, B., & Van Gorp, K. (2015). Determining the Scoring Validity Of A Co-Constructed CEFR-Based Rating Scale. Language Testing, 32(4), 1–21.
Earl, L.M. (2003).Assessment as Learning. Thousand Oaks: Corwin Press.
Eckes, T. (2009). Many-Facet Rasch Measurement. In S. Takala (Ed.), Reference Supplement To The Manual For Relating Language Examinations to the Common European Framework of Reference for Languages: Learning, teaching, Assessment, 2–54. Strasbourg, France: Council of Europe/Language Policy Division.
Falchikov, N. (1986). Product Comparisons and Process Benefits of Collaborative Peer and Self-Assessments. Assessment & Evaluation in Higher Education.11, 146–166.
Figueras, N. (2012). The Impact of the CEFR. ELT Journal, 66(4), 477–485.
Fulcher, G. (2010). The Reification Of The Common European Framework Of Reference(CEFR) And Effect-Driven Testing. Advances in Research on Language Acquisition and Teaching, 15–26.
Fulcher, G., Davidson, F., & Kemp, J. (2011). Effective Rating Scale Development For Speaking Tests: Performance Decision Trees. Language Testing, 28(1), 5–29.
Glover, P. (2011). Using CEFR Level Descriptors To Raise University Students’ Awareness Of Their Speaking Skills. Language Awareness, 20(2), 121–133.
Hand, D. (1996). Statistics and the Theory of Measurement. The Royal Statistical Society, 159(3), 445–492.
Huhta, A., Alanen, R., Tarnanen, M., Martin, M., & Hirvela, T. (2014). Assessing Learners’ Writing Skills In A SLA Study: Validating The Rating Process Across Tasks, Scales And Languages. Language Testing, 31(3), 307–328.
Hulstijn, J. H. (2007). The Shaky Ground beneath the CEFR: Quantitative and Qualitative Dimensions of Language Proficiency. The Modern Language Journal, 91(4), 663–667.
Hulstijn, J. H., Schoonen, R., de Jong, N. H., Steinel, M. P., & Florijn, A. (2011). Linguistic Competences Of Learners Of Dutch As A Second Language At The B1 And B2 Levels Of Speaking Proficiency of the Common European Framework of Reference for Languages (CEFR). Language Testing, 29(2), 203–221.
Ibberson, H. (2012). An Investigation Of Non-Native Learners €TM Self-Assessment Of The Speaking Skill And Their Attitude Towards Self-Assessment. (Doctoral dissertation, University of Essex, UK). Retrieved from British Library E-Theses Online Service
Idris, M., & Zakaria, M. H. (2016). Gauging ESL Learners ’ CEFR Ratings on Oral Proficiency in Rater Training. Man in India, 96(6), 1675–1682.
Jones, N., & Saville, N. (2009). European Language Policy: Assessment, Learning, and the CEFR. Annual Review of Applied Linguistics, 29(2009), 51–63.
Kang, O. (2012). Impact of Rater Characteristics and Prosodic Features of Speaker Accentedness on Ratings of International Teaching Assistants’ Oral Performance. Language Assessment Quarterly, 9(3), 249–269.
Linacre, J.M. (2014). Winsteps Rasch Measurement Version 3.71 [Software]. Retrieved from: http://www.winsteps.com.
Little, D. (2006). The Common European Framework of Reference for Languages: Content, purpose, origin, reception and impact. Language Teaching, 39(3), 167.
Little, D. (2007). The Common European Framework of Reference for Languages: Perspectives on the Making of Supranational Language Education Policy. The Modern Language Journal, 91(4), 645–655.
Lodico, M., Spaulding, D. &Voegtle, K. (2006). Methods in Educational Research: From Theory to Practice. San Francisco, CA: Jossey-Bass
North, B. (2014). The CEFR in Practice. English Profile Studies, 4. Cambridge: Cambridge University Press.
Nulty, D. (2011). Peer and Self-Assessment In The First Year Of University. Assessment & Evaluation in Higher Education, 36(5), 493–507.
Remler, D. K. & Van Ryzin, G. G. (2015). Research Methods in. Practice: Strategies for Description and Causation. Thousand Oaks, CA: Sage.
Schoenfeld, A. H. (2016). 100 Years of Curriculum History, Theory, and Research. Educational Researcher, 45(2), 105–111.
Topping, K. (1998). Peer Assessment Between Students in Colleges and Universities. Review of Educational Research, 68(3), 249–276.
Tyler, R. W. (1949). Basic Principles of Curriculum and Instruction. Chicago: University of Chicago Press.
Valax, P. (2011). The Common European Framework of Reference for Languages: A Critical Analysis Of Its Impact On A Sample Of Teachers And Curricula Within And Beyond Europe (Doctoral dissertation, University of Waikato, New Zealand). Retrieved from http://researchcommons.waikato.ac.nz/handle/10289/5546
