This report evaluates a development project designed by School 21 and the University of Cambridge to improve the language skills of Year 7 students. What further development of the Oracy Curriculum, Culture and Assessment Toolkit is needed and would enable a more robust evaluation of its impact.
Introduction
What was the impact of the pilot on the development of students' speaking skills. The Language and Literacy Research Group at Sheffield Hallam University contributed to the Voice 21 website review.
Methodology
A key feature of School 21's approach to orality is the development of a whole school culture of oratory. In addition, the evaluation team met with the University of Cambridge team to review the use of the Oracy Evaluation Toolkit and the work being undertaken by the team to assess the validity and reliability of the evaluation tasks.
Findings: Impact
There is a less clear but positive indication of progress at School 21 in relation to the paired teaching task. The ratings for School 21 were higher for all the follow-up assessments than those of the comparison school where progress rates were lower than for School 21.
Findings: Oracy Skills Framework
For example, School 21 told us that they initially focused on the cognitive side of the framework, but teachers discovered the need to develop the physical side (pitch, eye contact, etc.) and this led to a focus on performance poetry. The framework is also used explicitly with students to support self- and peer-evaluation. We have tried to infuse the four components into every aspect of the work. When they draft and rephrase [the Ignite speech], they coach each other. The children can explicitly say the area they are thinking about. where they are strongest and in the area they think they are weakest.
The framework is used to inform approaches that engage students to consider what might be appropriate in different contexts. The framework is designed to support a recursive curriculum, reviewing different skills in relation to increasingly challenging content and contexts, where context is understood in terms of the purpose of the conversation and the type and size of the audience. The framework appears well designed to inform the development of an orality curriculum that will support progress in the lessons.
At School 21, the framework appears to have supported staff in developing a broad speaking skills curriculum for their students, to provide a variety of purposeful contexts for conversations. Furthermore, in foregrounding the framework in the presentation of the curriculum, there is a risk of overlooking School 21's emphasis on providing diverse contexts for oration.
Findings: Dedicated Year 7 curriculum
Support for the development of conversation for formal presentational purposes appears to be a strong point of the curriculum. During the student focus groups, students spoke confidently about the different strands of the framework, but did so primarily in terms of introductory speech. Teachers talked about the value of group discussion, for example, and we saw many examples of opportunities for discussion and teachers providing support for students to engage in group discussion.
Although these are elements worth exploring, this suggests that the cognitive strand is being understood in terms of imparting knowledge with less emphasis on exploratory talk. Given the developmental nature of the project, we are unable to reach a definitive assessment of the effectiveness of the dedicated Year 7 oracy curriculum. The distinctive quality of the dedicated Year 7 curriculum is the identification of a set of skills which can be used across different contexts, with an emphasis on making appropriate choices about how to draw on these appropriately in these different contexts.
This would require an explanation of the value of both formal and informal opportunities to talk, and support for schools in providing opportunities for students to use talk to learn, i.e. addressing this will involve looking deeper concentrate on the cognitive component of the framework.
Findings: Oracy in every lesson
Creating a climate in which conversation is valued would certainly seem to be an important condition for exploratory conversations to flourish. While these reflections were helpful, the emphasis on presentation raised questions for the evaluation team about the place of exploratory conversation. They may have been less supportive of the freer exploratory conversation described in the opening of this report.
It is therefore not possible to draw conclusions here about the quality of exploratory talks across the whole school, but it is worth noting that this may be an area for further investigation, particularly in light of comments on the curriculum in Section 5.1 . The emphasis on meaningful contexts for speaking aligns with a focus on meaningful contexts for learning across the whole curriculum at school. Teacher interviews and lesson observations indicated that the commitment to promoting discourse across the curriculum in every lesson is a strong feature of the School 21 approach.
While students appear eager to engage in discussion and debate the evaluation team, we are unable to draw conclusions about the quality of exploratory talk (as we were only able to conduct a small number of observations). It seems that the development of exploratory talk is worth further investigation, especially in light of the results from the Raven's Matrics tests and our evaluation of the curriculum outlined in Section 5.
Findings: Whole school oracy culture
The development of proverbs is closely related to other aspects of the school curriculum and pedagogy. Other weights in School 21 on e.g. well-being, management and investigations are seen as both reinforcing and beneficial to the curriculum. It is also acknowledged that School 21's size and the whole school's speech culture make positive engagement far more likely.
It is also worth noting the reciprocal relationship between the public speaking curriculum and other provisions in School 21, where the development of public speaking is closely related to other aspects of public speaking. School 21's position as a new school with a small population may also mean that positive engagement is more likely as staff and students join a school where speaking is emphasized from the start. These school-specific factors raise questions about the extent to which the School 21 approach can be transferred to other settings.
However, its success may depend on several aspects of the very specific conditions at School 21. Any further evaluation of scaling up the intervention will need to consider how School 21's oracy approach is interpreted in other schools, and on opportunities and barriers, that arise when these approaches are implemented elsewhere.
Findings: Oracy Assessment Toolkit
The reliability of the Oracy Assessment Toolkit was independently assessed at Sheffield Hallam University through a teacher workshop. For each task, teachers were given an overview of the task and instructions produced by the University of Cambridge team and were also used in their testing of the Oracy Assessment Toolkit. Data from the Cambridge team's evaluation of the Oracy Assessment Toolkit suggest that there is variability in validity across specific tasks.
There is also a link to a document containing a glossary of skills highlighted in the Oracy Skills Framework. This would help in understanding the function and usefulness of the Speech Assessment Toolkit. The following section of the website provides an overview of the Oracy Assessment Toolkit tasks.
There is good evidence from the University of Cambridge team that they take the reliability and validity of the Oracy Assessment Toolkit very seriously. The team also recognizes that proper training is vital to the reliability of the Oracy Assessment Toolkit.
Supporting CPD and resources
The structure of the site reflects the elements of the Oracy Curriculum, Culture and Assessment Toolkit ie. the use of the Oracy Skills Framework to guide the curriculum and assessment could also be usefully highlighted. Drawing on some of the rationales presented in School 21's final report (Fidoe, 2014) would begin to address this issue.
It would be useful to more clearly foreground and justify the key elements of the Oracy Curriculum, Culture and Assessment Toolkit, i.e. in this regard it would be useful to explicitly identify the sources that informed the development of the curriculum make. It is currently unclear why these sources were chosen and whether (and if so how) they informed the development of the curriculum.
The website includes sections linked to each component of the Oracy Curriculum, Culture and Assessment Toolkit, although the relationships between these components could be more effectively indicated. The effectiveness of the proposed CPD package should be evaluated when it is rolled out to other schools.
Conclusions
Below we provide conclusions regarding the key components of oracy Curriculum, Culture and Assessment tools. It is important to note that the components are highly interrelated and that, when used in combination, they reinforce each other in terms of speech development. Findings from our qualitative research show that the oracy skills framework provided an appropriate and effective structure to support curriculum design and to support the development of a speaking assessment toolkit. A commitment to encouraging speaking in every lesson is a strength of the School 21 approach, as is the emphasis on providing meaningful contexts for speaking throughout the curriculum.
Further piloting of the Oracy Assessment Toolkit with the suite of support materials produced in this pilot project would allow the suitability of the 7-point scale to be explored along with further testing of interrater reliability. The evaluation highlighted the need for teachers to receive training before using the toolkit and for some reorganization of the Oracy Assessment Toolkit website. The proposed CPD package to support other schools in the implementation of the Oracy Curriculum, Culture and Assessment Toolkit has not yet been tested.
Putting boundaries on the intervention took time and it was only towards the end of the project that clarity emerged about the definition of what was now called the Oracy Curriculum, Culture and Assessment Toolkit. Pilot work would also provide the opportunity to assess the effectiveness of the proposed CPD package and the Voice 21 website, and further refine and test the Oracy Assessment Toolkit to determine reliability and validity.
School 21 and the University of Cambridge Oracy Skills Framework
School 21's mapping of the dedicated oracy curriculum to the Oracy Skills
Parental information sheet (intervention)
Opt-in parental consent form (intervention)
Information sheet control (school)
Information sheet control (parents)
Opt-out control school consent (parent)
Control school consent (Senior leader)
University of Cambridge initial and end assessments rating schemes
University of Cambridge assessment task grading conversion to scores
