Evaluation & Review Framework

The "Boud and Prosser principles" formed the basis of an evaluation instrument referred to as the Evaluation and Redevelopment Framework (ERF). Its purpose is to identify ICT-based learning design exemplars that foster high quality learning experiences, and to determine whether these exemplars have potential to be redeveloped in a more generic/reusable form.

However, in order for the evaluation instrument to be applied successfully, the following issues needed consideration:

• The potential for a learning design to foster high quality learning could only be assessed by applying the Boud and Prosser principles to an actual implementation of a learning design. Thus, a form/questionnaire that requested all necessary information about a learning design implementation needed to be designed.
• There was need for a protocol to describe a learning design in a consistent and concise manner yet distill its essence.
• The process could only hope to evaluate the potential for an ICT-based learning design to foster high quality learning.
• There was a need to provide a mechanism to determine the suitability of a learning design for redevelopment in a generic/reusable form.

Two workshops were conducted early in the project (March and April 2001) to address the incorporation of these issues into the ERF and to formatively evaluate the revised instrument. The ERF subsequently underwent further refinement by the project team based on feedback obtained from expert reviews and a further two formative evaluation exercises were conducted. To date, the ERF has undergone eight revisions.

The final ERF instrumentation comprises three main instruments, all of which are available here in rich text format:

1. Learning Design Submission Form: completed by the designer(s).
2. Learning Design Evaluation Worksheet: completed individually by two evaluators.
3. Learning Design Evaluation Form: A team of two evaluators reach consensus and submit one evaluation report.

Information sought from the submission form includes:

• A description of the learning design in terms of the tasks, resources, support mechanisms implemented; duration of the learning design, discipline used for, number of students catered for, and positioning within the broader program of study for the learners.
• Planned learning outcomes.
• Learner profile.
• Assessment requirements.
• Information technology requirements.
• Delivery context.
• Research findings about the learning design.

In addition, all resources utilised by the learners were requested for submission.

The worksheet and evaluation form comprises eight questions. The worksheet explained how to complete the instrument and enabled the evaluators to make individual notes. The evaluation form (completed by both evaluators) served as the final evaluation report.