Monthly Archives: November 2023

Update on MetaSENse: Evidence of increasing evaluation of ‘what works’ for students with SEND

Posted on by
Reply

metasense-im1

In this article, Professor Jo van Herwegen and her team give an update on the latest findings from the MetaSENse project which is revisiting the evidence base for effective interventions for students with Special Educational Needs and Disabilities (SEND).

Background of the wider MetaSENse project

The number of pupils identified with Special Educational Needs and Disabilities (SEND) continues to rise (DfE, 2021). Educational outcomes for those with SEND are often lower compared to those without SEND and this gap has become larger since 2020. This is likely due to the COVID-19 pandemic (Tuckett et al., 2022) and highlights the disparities for this population. Thus, it is important for parents, educators, specialist professionals and policymakers to understand the best evidence-based practice to raise educational outcomes in pupils with SEND.

The CEN’s MetaSENse[1] study, funded by the Nuffield Foundation, is synthesising evidence of what remediations work across different pupils with SEND aged 4 to 25. Technically, the project is focusing on “manualised” (i.e., has a published and accessible manual) targeted intervention approaches (either Tier 2 or Tier 3) that go beyond good quality teaching. Tier 2 interventions are often provided in small-group sessions in the classroom during independent work or during times that do not conflict with other critical content areas. Tier 3 provides intensive intervention sessions for individual students with more significant needs or whose needs are not sufficiently met by Tier 2 supports.

In phase 1 of the project, the team is carrying out a systematic review of the empirical literature, followed by a meta-analysis of the data. In addition to analysing of the quality of the evidence base, this meta-analysis will, for the first time, inform which Tier 2 and Tier 3 manualised interventions work best (that is, have largest effect sizes) in relation to different phases of education (preschool, primary, secondary, post-16), and in different educational contexts (special vs mainstream). And this is being done for each category of SEND needs. In phase 2, the team is using in-depth interviews with educational professionals to identify the barriers they face in implementing the most effective practices indicated by the aforementioned evidence.

The project will have a practical outcome: we will produce a toolkit featuring a database that can inform practitioners about the evidence-base underpinning different interventions for pupils with SEND, and which interventions to select in different contexts according to pupils’ needs. This will allow parents, educators, specialist professionals and policymakers to make evidence-informed decisions about how to raise educational outcomes for those with SEND in cost-effective ways. By the same token, it will inform the future research agenda of academics and relevant funders.

Update on our findings: the number of RCTs and QeDs included in MetaSENse

Randomised Control Trials (RCTs) are seen as the ‘gold standard’ way of evaluating what works. In RCTs participants are randomly assigned to one of two groups: the experimental group receiving the intervention or the control group which either receives the business-as-usual support in the classroom or another type of activity (named active control trial) that is not of interest.

Quasi-experimental designs (QeDs) are studies in which two groups of subjects are matched based on one or more characteristics and one group receives the intervention, whilst the other does not and receives either business as usual or an active control intervention. The difference with RCTs is that in QEDs the groups are not randomly allocated.

Together, these two types of study form the best kind of evidence that tell us how effective interventions for SEND are, in what populations and in what contexts.

In our systematic review, we began by collating all the research papers that reported the outcome of evaluating interventions to improve educational abilities in children with SEND. Using our pre-registered search protocol, we identified 55,564 records for title and abstract, which we then screened to evaluate their relevance. We then screened 4323 full texts, as well as full texts from clearing houses, organisations which write composite reports of evidence. From this set, we identified 533 records of studies that meet our inclusion criteria for the systematic review: over 500 studies to review! This initial work demonstrates that there are now a large number of studies that have examined Tier 2 and Tier 3 manualised interventions to improve educational outcomes for those with SEND.

How did we decide which studies to consider? The studies we considered were all published between 1st of January 2000 and 27/02/2023. We only included RCTs and QeDs that were published in peer-reviewed journals or dedicated websites of clearing houses and charities. Student dissertations held by universities were not included. Studies had to include a manualised intervention and report at least one educational outcome related to maths, reading, writing, science or overall attainment. We only included studies that focused on individuals with an existing diagnosis, or if the study screened for a diagnosis using normed assessments. Studies that only included students at risk for SEND based on teacher report or general attainment outcomes were not included, because this might target a much more heterogeneous population. Finally, studies could be completed in any country as long as the text was available in English.

What have we found so far? As can be seen from Figure 1 below, there has been a steady increase in the number of studies that have evaluated which interventions work to improve educational outcomes for students with SEND. Despite the steady increase in study numbers, it is important to note that this represents research globally.

The next step is to extract findings from these studies and use statistics to characterise the overall patterns – an analysis of the analyses, otherwise known as a meta-analysis. We have only completed data extraction for 25% of all studies and so far, have identified relatively few studies that have been carried out in the UK. In addition, the number of RCTs and QeDs alone does not yet tell us anything about the quality of the evidence and whether this has improved over time. So, with data extraction and quality analysis of more than 350 studies still to go… watch this space!

Our interim finding, however, is that there is an encouraging increase over time in the number of studies applying the best evidence-based approaches to evaluating the effectiveness of educational interventions for children with SEND.

Figure 1. Number of studies per publication year for MetaSENse:  All studies include RCTs or QeDs related to improving educational outcomes for those with Special Educational Needs and Disabilities.

metasense-im2

 

More about MetaSENse

You can find out more about the MetaSENse study and research team here: MetaSENse

The metaSENse study is funded by the Nuffield Foundation: The Nuffield Foundation is an independent charitable trust with a mission to advance social well-being. It funds research that informs social policy, primarily in Education, Welfare, and Justice. It also funds student programmes that provide opportunities for young people to develop skills in quantitative and scientific methods. The Nuffield Foundation is the founder and co-funder of the Nuffield Council on Bioethics, the Ada Lovelace Institute and the Nuffield Family Justice Observatory. The Foundation has funded this project, but the views expressed are those of the authors and not necessarily the Foundation. Visit www.nuffieldfoundation.org

 

[1] Raising educational outcomes for pupils with SEN and disabilities (MetaSENse)

References

DfE, June 2021: https://explore-education-statistics.service.gov.uk/find-statistics/special-educational-needs-in-england#releaseHeadlines-dataBlock-tables

Tuckett, S. et al., (2022). COVID-19 and Disadvantage Gaps in England 2021. Education Policy Institute, https://epi.org.uk/

“Did ChatGPT just ruin education?”

Posted on by
Reply

dalle_university-administrator

In this blog, Michael Thomas discusses the potential impact of generative AI tools such as ChatGPT on education.

Generative artificial intelligence, such as ChatGPT, is a form of AI that can generate human-like text based on a ‘large language model’ – information extracted from what is out on the internet. It can write essays and summarise facts, it can give feedback on written work and Excel formulae. There are versions that can generate other types of content, such as images from text, or music. I used DALL:E to generate the above image in response to the text prompt “draw a photorealistic picture of a university administrator thinking very hard about artificial intelligence” (I added the text!). Together, generative AI represents an immensely powerful tool.

In education, one of the principal methods of encouraging conceptual learning, developing writing skills, and assessing knowledge, is to ask students to independently write essays. However, students are now increasingly using generative AI in their work (see, e.g., this recent article from the BBC: ‘Most of our friends use AI in schoolwork‘). This is causing concern among educators and parents alike.

For educators, generative AI represents a significant challenge. Can teachers no longer use essays as an educational tool? Has a principal form of assessment been lost? Generative AI is immensely powerful but it has limitations: it generates plausible, ‘high probability’ text, not necessarily factually correct text, and the content it generates can be biased based on what the AI has found on the internet. Are students using a tool that leads them astray?

Like search engines, generative AI cannot be uninvented. Instead, students should be guided on how best to use generative AI to support their learning. But right now, students frequently know more about what generative AI can do than educators.

CEN Director Michael Thomas recently attended a meeting of the All Parliamentary Party Group (APPG) on Artificial Intelligence at the UK House of Lords, convened to discuss the potential impact (for better or worse) of generative artificial intelligence on education. He wrote a report of the meeting for the education think tank Learnus. The 3-page report can be found here.

Here are the main points from the report of the House of Lords meeting:

1. No one was panicking that AI robots were going to take over the world – although everyone recognised the downside risks of generative AI (e.g., inaccurate and biased content, age-inappropriate content, commercial ownership, data privacy). Instead, the main focus was on opportunities.

2. Among experts, there was a diverse range of views expressed on what tools like ChatGPT mean for education – all the way from ‘that don’t impress me’ to ‘it’s a steppingstone to utopia’. Some thought it on a par with the introduction of calculators to maths class, or of search engines for researching essays and projects: a helpful tool, necessitating some tweaking of teaching practice, but not much more. Others thought it would fundamentally alter educational practices and was an opportunity to democratise education – a tool to provide support for all.

3. The kids currently know much more than the teachers – pretty much everyone agreed that the most important first step is to improve teacher literacy on generative AI, to understand what these systems can (and can’t) do, and to begin to think about how they may be used. Perhaps the most important take-home for teachers and students alike is that you’ve got to know the limitations of the technology.

4. Guidance is beginning to emerge – institutions are thinking hard about the educational impact of generative AI, and some guidance is beginning to emerge (e.g., from the UK Department for Education and from the Russell Group of UK universities). As an example, this term, I gave a lecture to university psychology students on how they might use ChatGPT as a tool in their essay writing. I let them know what the chances are of getting caught if they simply use it to write their assessments (given that universities use AI detection tools, and that ChatGPT essays are reasonably easy to spot for content experts); and I also told them the very mediocre mark they would likely receive for an AI generated essay even if they didn’t get caught – because ChatGPT doesn’t write great essays. Here’s a slide summarising some tips:

generativeai-and-essay-writing-tips

 

There are many ways generative AI can be useful in education: to suggest initial ideas, to give feedback on text, to help second language learners improve their writing, for checking and recommending Excel formulae or computer code.

There are inevitably pitfalls we need to avoid (mostly linked to ensuring that content is unbiased and factually true, and that creativity is not stifled – ChatGPT will encourage you to write just like everyone else on the internet!).

But the broad message should be a positive one. In the same way that the invention of search engines gave everyone unprecedented access to vast stores of human knowledge (but ‘knowledge’ not to treated uncritically), generative AI can empower learners. The search is on for the best guidance to allow students to realise the potential of this new tool and avoid its pitfalls.

Did ChatGPT just ruin education? No, it gave education a powerful new tool, but with an instruction manual yet to be written.