Photo by Marvin Meyer on Unsplash

It seems that any investment made in education is generally assessed in terms of the impact on improving outcomes for learners. Not an unreasonable assumption, since the fundamental purpose of schools is to ’cause learning’ for their students, and anything a school does should reasonably be assumed to be adding to that goal.

Consider, for example, the investment made in text books and physical resources such as art supplies, outdoor ed and phys ed equipment, science supplies etc. The rationale for investing in these things is generally based on the way in which they will increase the opportunity for learners to learn what they need to. Then there’s the investment in buildings – the focus on providing warm, safe and acoustically sound learning spaces is based on the assumption that these are most conducive to learning. And so on.

Generally there are ways of measuring the impact of these investments on learning. For example, a damp, cold, noisy learning environment makes it difficult for teachers and students to function at their best – impacting negatively on their health. Lack of particular text books, phys ed equipment or science resources will inevitably limit the opportunity to learn in those areas.

So what about our investment in IT? There are a lot of claims about the positive impact of digital technologies on learning outcomes – some are valid, some over-hyped and some we simply don’t know for sure – yet. Every now an then the Ministry of Education demands proof that its investment in digital technologies (particularly in teacher professional development) is providing improved outcomes for learners, and then there’s ERO who from time to time seek to demonstrate this from their meta-studies of schools across the country. So too, boards of trustees will seek to be assured that the investment they are approving is contributing to improved learner outcomes (although there are still some that are simply satisfied to know that their investment in IT is keeping them ahead of the school up the road).

For a number of years now this has been a question asked of me in the work I do – and I’ve developed a simple framework, based on a simple intervention logic, which seems to be helpful. The reason for an intervention logic is to identify the various steps and stages between the actual investment in technology and the improved learning outcomes, with each of these stages being an opportunity to assess impact as a contribution to the eventual impact on learning. Simply asking for a causal link between technology and learning is pretty futile without this understanding.

Just recently I found myself asked the question again in a project I’m involved with, so thought it worth sharing the framework in this blog post. It looks like this:

Ignore the vertical representation of the framework here – I’ve altered it to fit the dimensions of the post, normally it is representing from left to right, from the technology to the outcomes.

To unpack this a little more, each of the stages is explained…

At the centre of what any school does to improve learning is their curriculum and learning design. This is the way a school enacts its beliefs and values, within the overall framework of responsibilities at a national level. Things to consider here include the creation, storage, management and access to curriculum-based resources, much of which is digital now. A key issue around access involves the need for access from places other than school (e.g. home, local library etc) and with that comes questions about permissions and rights to use etc. It also means considering not simply ‘delivering’ content, but also fostering content creators and contributors. Consideration also need to be given to the design of environments teachers and learners can use to collaborate and share their work (with other students, with teachers, with parents etc.)

If such platforms and environments are designed and available, then there must be changes made in teacher capability. The potential value of a digitally enabled curriculum won’t be realised unless there is a significant mind-shift among teachers, and a demonstrated shift in capability to thinking ‘digital first’ across a range of ways they work with learners. This will likely mean changes in pedagogical approaches – with a greater variety of pedagogical tools and approaches being implemented. Greater evidence of teachers working in more collaborative ways will be important, as technologically enabled environments tend to break down the need for a ‘one teacher – one class’ construct, and teachers are able to work to their strengths as a part of a team. It is important to note here that any investment in PLD will only be effective if the teacher has regular and supported access to the particular technologies and technology-enabled environments as a part of her/his regular, day to day work, and that these don’t add to workload, but serve to make them more efficient and effective.

The overall impact of these things happening should therefore be a significant shift in the experience of learners. This may be seen in the extent to which learning is personalised through the application of UDL principles, and the ability to direct learners to content that is customised to meet their needs rather than the ‘one size fits all’ approach of the past. Technology-enabled systems and platforms are particularly well served to achieve this. Other tech-enabled features that enhance the experience of learners include things such as single sign on to school managed learning systems, content and digital portfolios etc., opportunity to collaborate with peers on and off campus, and with community ‘experts’ who can assist learning. The range of ways in which learning progress can be shared with parents/whānau is also important here, providing additional avenues of support for learners. A focus on the development of digital literacy and digital fluency is essential here – a recent report from the OECD that includes results of a survey about the level of digital fluency among school-ages learners reveals that there is no country in the Western World where at least half the student population is good at navigating the digital world. In NZ it is 30% (see image below).

If all of the above are in place, then it can be expected that leaners will be more fully engaged in their learning. Boredom and low levels of engagement are characteristic of learning environments where there is little or no opportunity to customise the learning to meet individual needs, or where the approaches taken do not align with the particular interests or dispositions of learners. A pedagogically sound curriculum supported by tech-enabled platforms and environments have been shown to provide the conditions for increased learner engagement. Basic measures such as time on task or completion and retention rates are used here – but so too are many other more granular ways of measuring engagement in learning.

Research suggests that engagement in learning is an important pre-cursor to the learning being effective and learning outcomes improving as a result. Schools have all sorts of measures in place for monitoring improvements in learning outcomes, but these can only be attributed to the intervention of a particular technology as the evidence at each stage explained above is identified and

What this intervention logic demonstrates is that there is not (and indeed can not) be a direct causal link between the investment made in a particular technology, but that by understanding what we might look for at each step along the way, and having specific measures in mind – as well as strategies for remediating any lack – we can build a better understanding of the impact that investment has. And we can be more confident about the likely impact of the investment made on improving outcomes for learners.

So, some overarching questions to guide the process might include…

• Is there a collaboratively owned and enacted set of beliefs underpinning our schools vision and mission that guides all decisions about investment, resourcing and evaluation of programmes in the school?
• Is this used specifically to assess the need for (and urgency of) investment in any/all forms of technology?
• Are there clear sets of evaluative guidelines that flow from this to allow for the gathering of evidence of impact at each stage through the process? How does this happen? Who is involved? How frequently does it occur?
• What forms of remediation are considered where a lack of performance is noted? Whose responsibility is this?
• What other factors need to be considered as influencing what happens at each stage – besides the specific contribution that the technology is making? (e.g. internal factors, external factors?) Are there ways in which further technology-based intervention might help here?

This is merely a high level overview. of the process, but hopefully useful enough to convey the main ideas. I have developed a number of tools and frameworks to support this process within the specific contexts I’ve worked in over past years, but too detailed and too context specific to share here.

Happy to hear how others approach this issue. Is there something missing? Does this work for all contexts? Feel free to add thoughts in the comments below.