Is the KS2 Reading SAT testing reading?
I started this series with this a blog outlining the main issues with the Key Stage 2 (KS2) Reading SAT, why it needs reforming and possibly how here.
For the sake of needing to start somewhere, let’s look at the issue of validity. Kime states a valid assessment a) tests what it claims to and b) provides information that is valuable or appropriate to its purposes.1
It would seem obvious that the KS2 Reading SAT is a test of reading.
What is Reading?
Reading has 5 key components:2
1) Phonemic Awareness
2) Phonics
3) Fluency
4) Vocabulary
5) Comprehension
Comprehension requires the other four components and the necessary domain or background knowledge.3
The National Curriculum English programmes of study for KS1 and KS2 cover the component parts but do not always make them statutory.
The teaching of reading in the National Curriculum is split into word reading, which covers phonemic awareness, phonics and fluency and comprehension, which covers vocabulary and comprehension.
Does the KS2 Reading SAT cover these component parts?
What is the KS2 Reading SAT testing?
According to the STA test framework and handbooks, it is testing the attainment targets in the English programme of study in the National Curriculum.4 However, it only refers to the Year 5 and 6 attainment targets.
One would assume, in so far as possible, it is testing the objectives we are expected to teach for reading comprehension.
But it’s not that simple.
To be fair to the test designers, this is not a task that has been made straightforward. The programme of study is wooly in places and the role of knowledge in comprehension is not as clearly outlined as it should be.
The specific knowledge in the comprehension part of the programme of study was kept to a minimum (unlike for word reading). This was to allow schools the freedom to develop their own school curriculum.
How a national test could be devised to assess comprehension based on locally based curricula was not addressed.
While the STA’s job has undoubtedly been made difficult, they have not attempted to design a test based on the evidence on generic reading comprehension skills or the role knowledge plays in comprehension. This should have guided their rationale when selecting attainment targets to assess but it didn’t.
Instead, as Ofqual point out:
“Whereas the Test Framework for mathematics is essentially a re-presentation of the national curriculum framework, the Test Framework for reading is more of a re-interpretation. Indeed the eight strands also share some continuity with the seven assessment focuses previously used by STA to represent the content domain for earlier national curriculum reading tests, to support test development and marking.”5
In short, the STA have recreated the old assessment in so far as it can and continues to use questions from the item bank they set up in 2012. This has allowed the belief that comprehension is composed of a set of generic reading comprehension skills that can be transferred to different texts to continue to underpin test design and text selection. (I will return to the issue of the Reading SAT being untethered to the curriculum as well as the issues of breadth vs depth in a later blog).
The old assessment focuses (AF’s) were a mixture of generic reading comprehension skills (e.g. retrieval and inference) and evaluation of the text (e.g. writer’s choice of language, grammatical features and context of the text).6 The latter are more closely linked to secondary objectives for English as a subject (a mixture of English Literature and Language).
The revised KS2 Reading Test Framework includes five reading strands measuring reading comprehension skills compared to the previous two AF’s. There are three reading strands assessing evaluation of the text compared to previous four AF’s.7
How were the attainment targets that are measured selected and reinterpreted?
The validity frameworks for the Reading SAT include their rationale behind the attainment targets they don’t assess but not the ones they do.8
Clearly attainment targets such as ‘recommending books they (i.e.pupils) have read to their peers, giving reasons for their choices’ can’t be selected.
However, it is less clear why identifying themes and conventions can’t. They give no reason for themes, while they argue assessing conventions would require the teaching of particular genres of texts.
This ignores the fact that particular genres are expected to be taught e.g. myths. However it’s true that it’s not possible to assess conventions for other categories such as fiction from our literary heritage as these do not constitute a genre.
To digress for a short while, they claim that it’s too subjective to test the understanding of a wider range of fiction, poetry, plays, non-fiction and reference or textbooks. Yet the selection of texts for the Reading SAT is made from fiction, poetry and non-fiction. When attainment targets for both poetry and plays relates to performance not analysis, it is not clear why poetry is included and not plays other than their overriding concern to ensure continuity with previous tests.9
This concern explains their contradictory rationale – they don’t want to assess conventions because it would mean telling schools what genres to teach, yet the inclusion of poetry texts de facto means schools have to teach analysis of it even though it’s not an attainment target.
It also explains the arbitrary nature of the part of the attainment target they focus on. When they excluded themes and conventions they decided the attainment target was about maintaining positive attitudes.
Yet they divorce ‘explore the meaning of words’ from the overall target which involves (pupils’) checking the book makes sense to them and discussing their understanding.
From analysis of the vocabulary questions on the Reading SAT it’s clear that they thought they could make this attainment target fit their existing item bank. In doing so they have distorted the attainment target from exploring the meaning of words to knowing and giving the meaning of words which involves retrieving a word or deducing the meaning of a word (essentially guessing the meaning of the word from the context. This will be examined further in the next blog).
Why does it matter?
The attainment targets selected matter, as does the reinterpretation of attainment targets, as it determines the nature and scope of the assessment and its validity.
From 2003 to 2015, on average 63% of the marks were awarded for AF’s assessing reading comprehension skills in the Reading SAT.10 This has sky-rocketed since 2016. From 2016 to 2019, on average 94.5% were awarded for reading strands assessing reading comprehension skills.11
Clearly the Reading SAT is not valid, the selection and reinterpretation of attainment targets means it does not assess what it claims to. By ignoring the role knowledge plays in comprehension and focusing so heavily on measuring reading comprehension skills it does not provide valuable or useful information about the attainment targets or reading.
In the next blog I will focus on how the Reading SAT continues to drive the teaching and practice of generic reading comprehension skills at the expense of fluency, comprehension strategies and ultimately the goal of raising achievement in reading.
Footnotes:
1 Kime, S., (2017), Four Pillars of Assessment Validity (https://evidencebased.education/pillars-assessment-purpose-validity/)
2 https://fivefromfive.com.au/
3 The role of domain and background knowledge has been identified by multiple researchers including Hirsch, E.D., (2016), Why Knowledge Matters: Rescuing Our Children from Failed Educational Theories. Massachusetts: Harvard Education Press and Willingham, D.T., (2009), Why Don’t Students Like School? A Cognitive Scientist Answers Questions About How the Mind Works and What It Means for Your Classroom. San Francisco: Jossey-Bass.
4 2016 Key Stage 2 English Reading Test Framework – https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/628816/2016_KS2_Englishreading_framework_PDFA_V3.pdf
National curriculum test handbook: 2016 and 2017 Key stage 1 and key stage 2 https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/670132/NCT_Handbook_final.pdf
National curriculum test handbook: 2018 Key stage 1 and key stage 2 https://www.gov.uk/government/publications/2018-national-curriculum-test-handbook
National curriculum test handbook: 2019 Key stage 1 and key stage 2 https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/868425/2019_national_curriculum_test_handbook.pdf
5 Content validation study: 2016 key stage 2 reading and mathematics tests https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/649573/Content_validation_for_mathematics_and_reading.pdf
6 Assessment Focus 1 – Use a range of strategies including accurate decoding of text, to read for meaning – was not assessed in the KS2 Reading Test. https://webarchive.nationalarchives.gov.uk/20090327072110/http://www.standards.dfes.gov.uk/primaryframework/assessment/app/waaf
7
8 Appendix F: 2019 Validity framework Key stage 2 English reading https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/868437/Appendix_F_2019_validity_framework_key_stage_2_English_reading.pdf (pp. 5-7)
9 So far only poem has only been included in the 2018 Reading SAT paper.
10 The average of the percentage of marks awarded to AF2 and AF3.
11 The average of the percentage of marks awarded to Reading Strands 2a to 2e (vocabulary, retrieval, summarisation, inference and prediction). Vocabulary is not a generic reading comprehension skill but the kind of vocabulary questions asked expect pupils to retrieve or deduct meaning of the word.