Developing and Implementing an Evaluation Process for Organisational Capability in Technology Enhanced Programme Development
A two-year project, completed in 2018, to develop and implement an evaluation process for organisational capability in technology enhanced programme development in Tertiary Education Organisations. A collaboration of Ako Aotearoa and Synapsys. Participating Tertiary Education Organisations (TEOs) were CareerForce, The Skills Organisation Inc., Literacy Aotearoa, NorthTec, Otago Polytechnic, Taratahi Institute of Agriculture, Wellington Institute of Technology (Weltec) and Whitireia New Zealand.
The main aims of the project were to:
- better understand their organisation’s current capability to respond to technology in learning, measured against an external rubric
- build internal engagement and understanding of the challenges to be addressed
- identify opportunities for development that will make the biggest impact
- gain a better understanding of sector wide capability and benchmark themselves against it
- support the wider development of the sector.
The project methodology involved:
- participating organisations surveying their staff against a set of benchmark indicators
- evaluating the results in terms of each organisation’s current capability and potential opportunities for development
- all participating organisations participating in a joint process to explore opportunities at a sector level for capability development.
Dr Stanley FrielickAko Aotearoa
Helen LomaxAko Aotearoa
The key findings from the project included:
- The areas identified as providing the greatest opportunity for development were: content creation, capability development, programme design, course design, and delivery.
- A repeating theme was the need for new ways to keep engaged with the reality of learners. Challenges included: training needs analysis and learner profile analysis tools are useful, but can become mechanical and lose insight into what really matters to learners; when it comes to understanding bespoke or employer-specific programmes, there is a danger of promising outcomes or experiences that are not adequately informed around genuine learner needs; it can be unrealistic to get learners intimately involved in the development process.
- Several organisations were surprised about the spread of response in some key areas, such as: how well the vision for learning was understood across the organisation; how clear people were about the approach to using particular technologies; what progress was being made against strategic and development objectives.
- The scale of change many organisations are going through throws up large capability development challenges that touch on almost all aspects of activity, from high level programme design that incorporates thinking on new technologies, through to the skills required to deliver effectively in a blended learning model.
- Participating organisations recognised the need for new skillsets and new ways of working to respond to challenges with technology. However, simply hiring instructional design and project management capability is not, in itself, enough. Changes are as much around change management as they are pedagogy.
- In terms of meeting the five aims of the project the following shows the percentage of organisations that met objective: better understand current capability 63%; build internal engagement and understanding 72%; identify opportunities for development 81%; better understand sector capability and benchmark 72%; support the ongoing wider development of capability 100%.
- All participating organisations wished to continue involvement in an ongoing collaboration around capability development and want to extend the collaboration model beyond the end of the project.
The key recommendations, trends and implications from the project included:
Microlearning and credentialing | While qualifications and programmes will remain central to learning, there is a growing desire for more flexibility in learning pathways and recognition of progress at smaller, granular levels. Organisations are challenged to break learning up into smaller, bite-sized chunks that give learners more flexibility in the sequence of study. There’s also a flow-on expectation that learners can get recognition for completion of these chunks long before they add up to a course or qualification.
Less content development, more curation | There is a growing recognition that: the world is full of content that learners can be supported to access without organisations needing to redevelop it; organisations do not have the capacity to develop and maintain all content for all programmes; in the future, development will be more about setting up content specialists to work effectively with learners to curate and work with their own content.
Transfer of learning on the job and learning impact measurement | Employers are less interested in training activity and more interested in outcomes, that is, they want to know what learners are doing differently on the job that delivers value. Again, curricula and qualifications will remain important, but organisations are being challenged to reframe them around outputs and impact of learning.
Future implications | What does this mean for staff capability development in the future? The following are suggestions that emerged out of consultation that may be relevant: break up courses into smaller, bite-sized chunks of learning and develop them as stand-alone experiences; use instructional strategies that have learners building or adding their own repositories of knowledge.
Future projects | The following are being considered for a follow-on to this project: Collaborative piloting of emerging technologies that might support the above objectives; Conducting a follow-up evaluation to measure the success of current initiatives to build staff capability; The roll-out of the evaluation framework on a wider basis to support further targeted capability development initiatives.