ORIGIN OF THE LEARNING DESIGN
Prior to the development of this learning design, the designers had implemented other strategies in the subject to promote learning, namely, the use of case studies. Staff played the roles of users in organisations and students interviewed them. From the designers' perspective, such an approach worked well as students learned a great deal from the level of complexity inherent in it. The students however, felt that staff had too much power and expressed a preference for a different method of learning . This is how the idea to develop the simulation originated.

The desginers wanted to design learning experiences that would facilitate students' learning about the complexity of process as well as learning technical skills and how to apply them in an organsiational context. Importantly, the design team wanted students to develop an appreciation of the difficulties involved when designing and developing Information Systems and experience a level of uncertainty that accompanies such activities.

Penny Collings obtained advice from colleagues in the Psychology Department in her university regarding the development of the simulated workplace. Together they decided on a learning design that would enable students to experience the situations common to software-analysis work in the business world, by playing a variey of staff roles whithin an organisation, including those relating to IT.

The design team obtained funding from the former Australian Government Committee for the Advancement of University Teaching (CAUT), to assist and continue the development of this learning design.

Excerpt adapted from Alexander and McKenzie (1998, p. 156).

Reference:
Alexander, S. & McKenzie, J. (1998). An evaluation of Information Technology projects for university learning, CAUT, Canberra: Australian Government Publishing Service.

TIMES THE LEARNING DESIGN HAS BEEN USED
There have been about 15 CHA's created and run so far.

At the time of submitting this description, the simulation is not being run due to staff changes and the fact that it requires considerable energy; people need a break after a certain time in such intensive work.

However, it may well be used again in a few years and is ready to be used.

MODIFICATIONS SINCE FIRST USE
The main modifications have been in terms of:

Assessment: Students were first assessed for their reflection on the work they did. Gradually this changed so that they received marks for the actual work done. This was by student request and negotiation.

IT infrastructure: We gradually strengthened the role of the IT infrastructure in the game. At first we did not declare it to be central but, over time, we found that the benefits that students gained by creating an online organisation were important and central to the success of the game.

These are the two most important developments but we continually refined the game through experience.

DISSEMINATION
There has been no active dissemination, though it is certainly available. We have discussed the work with several interested people. I believe they get ideas and then use the ideas in a modified form in their own teaching.

Generally, I think this type of teaching/learning is running into some difficulties these days because it is so intensive and not suited to all students.

It is not easy to teach off-campus, it requires a dedicated group of people to run the game, and this type of teaching resource/support is not easily transported off-shore, for example. The same applies to the Lotus Notes infrastructure, though other environments (e.g. Sharepoint) could be substituted. The start-up effort is considerable and needs a sympathetic overall learning environment to allow this type of learning to be undertaken.

There have been huge spin-offs from this work around the university in the use of CSCW in general; the benefits are not solely in the reuse of the game itself.

RESEARCH CONDUCTED ON THE DESIGN
Evaluating this style of learning is difficult. Student evaluations vary according to how much distance they have achieved from the game. Although this is generally known, we have not had the resources to undertake longer-term evaluations (e.g., asking students to evaluate the experience six months or one year later). Asking students for their perceptions of the subject is conducted on the last day of semester but this is unrealistic. Feedback varies a lot and is partly dependent on the dynamics of the specific group (CHA).

Nevertheless, we have used feedback from students and staff to change and develop the game based on the experiences of each group each year.

Further, this simulation was evaluated by Alexander & McKenzie (1998). Comprehensive documentation is provided in:

Alexander, S. & McKenzie, J. (1998). An evaluation of Information Technology projects for university learning, p. 155-168. CAUT, Canberra: Australian Government Publishing Service.

We have also published extensive evaluation data in two University of Canberra Technical Reports and have published papers describing and evaluating some of our work. The Technical Reports contain extensive demographic data and data about the design and use of CSCW based on Lotus Notes. Use of computer-mediated-communication is assessed there. Feedback from student evaluations is also included. Copies of these reports are available on request.

The main focus/findings of evaluations conducted relate to:

How to use IT to supplement face-to-face meetings/good design of online workspaces:

The outcomes in relation to this are documented in these two Techncial Reports:

Walker, D. (1999). Analysis of Groupware Usage, 1997-8, University of Canberra TR 99/113.

Walker, D. (1997). Analysis of Groupware Usage, 1996, University of Canberra, TR 97/75.

Outcomes of Computer Supported Collaborative Learning:

We have undertaken several studies about this and reported the value of such collaborative work as collaborative learning. For example, refer to:

Collings, P. and Pearce, J. (2002) Remote web site usability testing: a collaborative approach between university students. British Journal of Educational Technology.

Another paper investigates the relative merits of teaching in the traditional manner with set problems (simple, tightly constrained); versus role-playing (by staff) as a way of having students focus on user requirements (gives a strong user focus and a more complex view of systems); versus role-playing (as in the CHA) by students as a weaker way of having students focus on user requirements (students in role seem to make weak clients) but as a richer way of them experiencing the organisational context of systems design. See:

Walker, D.W., Collings, P. Hicks, F., McMahon, A., & Martin, E. (1994). A comparison of methods for teaching information systems design, Proceedings from CHI '94, 245-246, Boston.

DESIGN EFFECTIVENESS VERSUS INTENDED OUTCOMES
We undertook evaluations at the end of some semesters to ask the students what they did learn. It turns out that this varies between students, with their inclinations and abilities and interests in learning, and their peers can have an influence on the learning outcomes of the whole CHA group too.

Many students gained enormously from this form of learning; many felt they did not learn "properly" because they disliked the uncertainty and the requirement for active involvement. The outcomes are reported at length in Walker (1997, 1999) (see above for citation).

One of the problems is that it is not appropriate to ask the question until a considerable time after such games have run. We have not had the opportunity to follow-up on this but other studies of behavioural simulations would indicate that they are a powerful way of learning.

There are a couple of papers that are particularly relevant to our work in this sense:

Akerlind, G. and Trevitt, C. (1995). Enhancing learning through technology:when students resist change, Proceedings, ASCILITE'95 - Learning with Technology (Melbourne)

Gamson, W.A. (1978). SIMSOC - Simulated Society (3^rd ed, Free Press, NY).

UNEXPECTED LEARNING OUTCOMES
By taking control of their own use of online workspaces to manage their own group projects, students have found them to be a powerful project management tool. If a student (in a group) says s/he has done some work, they are asked to come up with the goods by placing them online - no more excuses that they left them at home!

Students also gain generic skills in CSCW and transfer them to future projects. Students have also learnt that information management is a key issue in online workspaces - how can they find the relevant pieces of material in a space used by many students? Students learn, experientially, the need for information management. This was not foreseen but observed in practice.

We had to be willing to let things emerge. This meant that CSCW design was an evolutionary process. We all became aware of the significance of this - the ability to design then modify your own work practices - as a core element in understanding work and hence information systems design (IS design is about the design of work after all).

Another unexpected outcome was the fact that the Univeristy of Canberra became known as a source of support for shared group workspaces (through Lotus Notes). Several people and organisations have used this resource to support short- or long-term projects of an academic nature. This idea of having infrastructure that others can access was an interesting (informal) outcome of this work in general.

HOW LEARNER ENGAGEMENT IS SUPPORTED
Are learner expectations identified and built upon?
Yes - by feedback from the learners. We take this into account in assessment and the students themselves work together to define their own tasks.

Are learners' prior experiences taken into account?
We like to think so. We have students from several courses and cultures who come together in this simulation. This is intentional so that we get a good mix of backgrounds and hence perspectives on systems. The major problem is their lack of experience of any kind of role-play/gaming and the fact that not all staff think this is a useful way of learning (this can sometimes undermine such games).

Do learners experience key concepts in multiple ways?
They certainly do. One approach to this is to have students change roles during the game. They then get two points of view at least, plus they see the work of their peers on the same project. Further, we ask student groups to reflect on what they are doing and report back to the whole class. This gives them further perspectives.

Is there opportunity for peer interaction and feedback?
Yes, see above, plus students are given feedback on their portfolios by staff.

Do the assessment tasks support engagement?
The engagement is what is assessed.

Are learners encouraged to reflect on their learning experience?
Yes, see above.

Are learners given a sense of control in conducting the activities?
Yes, they have a lot of influence in defining the activities. Sometimes this is difficult because they can't imagine what is to be done (at least initially). However, we give support for this in the Participants' Manual.

Does the learning experience engage students affectively?
Yes! They get very involved both affectively and also effectively.

Comment from the Evaluation Team...

The results from the project's evaluation of this learning design exemplar indicate that learner engagement is supported very well. Below is the summary provided by the evaluators:

"The design is based on an experiential learning setting. Learners are very much required to use their prior experience and expertise in the roles that they undertake. The key concepts of systems design are experienced through completion of relevant tasks in conjunction with peers and with tutor feedback. The assessment of the learning is based on the development of a product and the use of a reflective journal mapping the learning experience. The student assumes several roles in the learning process and is required to reflect in organised ways on a number of aspects of the learning activity. There is a large degree of freedom and choice in the roles assumed and the nature of the activity undertaken in the role."

ACKNOWLEDGMENT OF LEARNING CONTEXT
Do the activities link both specifically to the field of study/professional practice and consider the broader context (such as social, political, economic, and environmental) circumstances?
Yes, that's exactly the purpose of the subject and the learning activities - to simulate a slightly futuristic, organisational environment and the development of IT plans and information systems to support the work of that organisation.

Do assessment tasks match the intended learning outcomes?
Yes, after considerable negotiation, they do so in a reasonable manner. All work done for the CHA is assessable and students are also required to reflect on the work done.

Does the learning design assist students to see how their learning can be used in other situations than the ones given?
Yes, especially because students develop considerable generic skills related to CSCW, group work in general, project management, quality management, planning, negotiating, communicating, etc. The techniques of prototyping information systems and user interfaces and evaluating them is common to the development of all computer-based systems so is transferable. Understanding the complexity of the workplace environment and organisation dynamics is also important. Seeing the same environment from several points of view (through role-taking) is also important.

Are there cultural assumptions built into the learning design?
The idea is that a "Cultural Heritage Authority" could exist in any country; it is not specific to Australia. However, given that it has only been used in Australia, and given that the classes have students from many cultures, the question is important. In fact I feel that, as there is an increasing percentage of students from cultures that have basic learning models that are significantly different from experiential learning, the behavioural simulation is more problematic. If students are used to rote learning and learning from a "master", then a behavioural simulation is rather challenging and difficult to come to grips with. This applies to many students, not just those from off-shore. We have not tackled this problem and have not run the game since the balance of students has changed. It is worth seriously addressing but would take a lot of resources, I suspect, and considerable preparation time to introduce different learning approaches to students before the start of such a game.

Comment from the Evaluation Team...

The evaluators concluded that the learning design acknowledge the learning context effectively. The following feedback was provided:

"The learning setting is contextualized and attempts to make the connection to the field of practice strong and meaningful. The setting is very dependent on students applying information-systems concepts within the tasks they undertake. There is a high degree of constructive alignment between the objectives, the learning activities and assessments of the learning. The learning setting provides learners with access to a variety of resources needed to support their activities and the roles and responsibilities are clearly articulated to ensure the process is achievable. Resources include information systems similar to real world settings."

HOW THE LEARNING DESIGN CHALLENGES LEARNERS
Are students given the opportunity to question their knowledge and experience thus becoming self-critical of the limits of their knowledge base and their assumptions?
I think so. In fact they feel quite challenged in attempting to come to grips with this relatively new (for them) learning environment. There is a high level of uncertainty, the students must interpret and adapt their theoretical knowledge to a new learning environment that is challenging and also an opportunity. When they submit their portfolios, they have the opportunity to reflect on the way they adapt.

Does the learning setting assist students to go beyond the resources provided for them?
We provide a library to students - lots of reference material about cultural heritage and tourism matters, educational settings for the work of the CHA, etc. We also provide "interview" transcripts with people who represent potential clients or stakeholders in the CHA. Students are encouraged to do research to ground their designs. Some do so.

Are students able to make decisions about planning, directing and assessing their own learning?
Yes, as discussed above, though they don't assess their own learning in any formal manner. Students don't mark their own work or that of others, for example, but they do reflect on their own work and provide drafts to their peers as part of the quality management processes within the CHA.

They also run design and project review sessions so to that extent they assess their work and work is learning (I believe) or at least it is often difficult to separate the two - that's one of the assumptions of this behavioural simulation.

Comment from the Evaluation Team...

The evaluation concluded that this learning design exemplar challenges learners effectively and offered the following summary:

"The design requires students to immerse themselves in a field of practice. The tasks they must complete are open-ended and complex and provide a degree of challenge to them. The use of a large anchoring task provides a solid basis for the learning activity and the requirements of the task enable learners to determine the scope and extent of the sills, knowledge and understanding required to complete the task and complete the required learning processes. The activity involves a high degree of reflection in its conduct and this reflection encourages self-criticism and metacognitive activities as do the interactions with other key players in the activity."

OPPORTUNITIES FOR PRACTICE
Are students encouraged to articulate and demonstrate to themselves and to others what they are learning?
Yes, they present their work informally on a weekly basis within the CHA, plus we have several more formal occasions when students present within the CHA or to the whole class. This is one way of illustrating what you are learning. The other way is through the entire online workplace that is the CHA. All material (that students develop as members of the CHA) is available there for students to share with each other.

Is sufficient practice provided to enable expertise to be realised?
Possibly not. Many of the core skills required are developed in previous subjects. The design of this game is as a capstone subject. However, the requirement is that some new skills are developed, but students can avoid this depending on the roles they take. This is a disadvantage of the game, but we believe this is balanced by the other advantages.

Does the learning design help students to apply criteria that indicate they are learning appropriately?
I'm not really sure about this. Certainly the assessment schedule gives students a framework for considering this. Also, the debriefing and briefing sessions are intended to help students keep on track and keep focussed on the aims of the whole game, and to provide new learning material to help develop the new skills and understandings and processes they will require to be successful participants in the game.

Is appropriate feedback available at key points in the learning process?
Yes. We tried several ways of doing this. We also offer individual debriefing sessions for students. Sometimes these are required but now they are voluntary. Other feedback is as already discussed, e.g. group debriefing sessions plus feedback on their two portfolios (the first is early enough for this to be helpful for the rest of the game).

Is there a clear alignment between the activities conducted and how the students are assessed?
Yes.

Comment from the Evaluation Team...

The evaluators stated that practice is supported well by this learning design. They noted:

"The experiential design is very much built around the concept of practice. Students are immersed in very practical activities that require high degrees of self-regulation and reflection while support is provided by a number of accompanying features. From a practical perspective, the design is very much one where students model the real world setting where the skills are required to be applied. Communication and demonstration are key points and the practice is supported by access to underpinning knowledge through the information base that is provided."