PEDAGOGY AND QUALITY ASSURANCE IN THE DEVELOPMENT OF ONLINE LEARNING FOR ONLINE INSTRUCTORS

Terry Cowham
The Open University in the North West

Julia Duggleby
The Sheffield College, UK

ABSTRACT

The last six years have seen an increasing interest in the use of online technologies for delivering training in the UK, as employers recognize that these technologies can offer a more economic and flexible training package that can be tailored to the priorities of a company and its employees. This article explores the development of the Certificate in Online Learning course, a course developed by an academic organization, The Sheffield College, in partnership with the Chartered Institute of Personnel and Development. The article considers the way in which the strengths and experience of the partners have been brought together to build a new and robust course, based on sound pedagogical principles, that addresses the needs of corporate trainers in the development and delivery of online training. In addition, the article examines the role of the Open College Network, the accrediting body for the course, in ensuring its continued development.

KEYWORDS

Online Corporate Training, Online Teacher Training, Corporate-Academic Partnerships

I. the CIPD and the development of the certificate in online learning (COL)

The Chartered Institute of Personnel and Development (CIPD) is the professional body for those involved in human resource management in the UK. The CIPD has more than 18,000 members throughout the UK and Ireland, and asserts that “a qualification from us is the passport to a career in people management and development and is highly valued by employers” and that “we are committed to developing and promoting best practice to meet the ever-changing needs of our profession and our members. We achieve this by defining and promoting levels of competence through our Professional Standards (http://www.cipd.co.uk).

The CIPD recognized that e-learning has the potential to revolutionize the way we learn but that the arrival of such a radical new approach inevitably brings with it uncertainties about the most appropriate strategies to adopt. This recognition led the CIPD to ask The Sheffield College to collaborate in developing the Certificate in Online Learning (COL) to provide students with a thorough understanding of how online learning works within the workplace, its advantages and disadvantages, and the skills required to commission and manage online learning programs. Delivered entirely online with extensive support from both teacher and peers, the program provides each student with the unique experience of being both a learner and an instructor. The certificate program was developed from a partnership that combined the established expertise of the CIPD in delivering courses for trainers and human resource professionals with the experience and skills of The Sheffield College in designing online learning.

II. The Origins of COl

A. Living IT

The Sheffield College's experience in online learning began with the development of a small suite of web-based learning programs called Living IT. Living IT was conceived in the summer and autumn of 1995 through a partnership of The Sheffield College (http://my.sheffcol.ac.uk) with Fretwell Downing Data Systems (http://www.fdgroup.co.uk/), David Jennings Associates (http://www.djassociates.com/intro.html), and Manchester College of Arts and Technology (http://www.mancat.ac.uk). The aim was to use the web as a medium for learning about the web; the rapid evolution and volatility of the web required a pedagogic strategy that would cope with rapid evolution, which would make print-based materials inappropriate and redundant.

A key design feature of Living IT, developed in 1996, was its accreditation using the credit-based framework of National Open College Network's (NOCN) (http://www.nocn.org.uk/), which requires course teams to express learning in terms of outcomes, with explicit assessment criteria, before embarking on a pedagogic design and the preparation of learning materials.

It soon became clear that online learning placed demands on teachers that were different from those encountered in face-to-face (F2F) tuition. One instructor said, “I found myself having to make up the rules as I went along; it was like a tar baby….I felt disembodied, disadvantaged, and disempowered…a combination of driving instructor and baby sitter [1].” In fact, a significant proportion of teachers were likely to be more fazed by the challenge of online learning than their learners were [1].

B. Learning To Teach Online (LeTTOL)

In 1997, when the Further Education Development Agency (FEDA)—now Learning and Skills Development Agency (LSDA) (http://www.lsda.org.uk)—launched its Quality in Information and Learning Technology (QUILT) staff development initiative , The Sheffield College, in partnership with the other colleges in the South Yorkshire Further Education Consortium (SYFEC) (http://www.syfec.ac.uk/) successfully applied for funding to develop an online staff development package for online teachers called Learning to Teach Online, or LeTTOL.

The following key pedagogic principles were adopted from the start:

• Accreditation should use explicit learning outcomes and assessment criteria to inform course design and assessment strategies.
• All course activities and materials should be delivered over the internet using web technology.
• Team working and learning should underpin both course development and course delivery.
• Online learning sets should provide the model for team learning.
• Learners should use the web to learn about the web's potential as a learning medium and to evaluate its differences from face-to-face learning.
• Before teaching online, an instructor should first have experienced the course as a learner.

Since its launch in 1997, the LeTTOL course has been delivered to more than 1,500 learners and has grown both in scale of delivery and in reputation. The majority of its learners are based in the United Kingdom, but many have come from as far afield as Australia, Chile, Canada, and South Africa.

What's more, the LeTTOL course and its instructors, have been recognized on many occasions by various awards. These include the following:

Table 1. Awards for LeTTOL and its Instructors

III. The Certificate in Online Learning (COL) program

The success of LeTTOL meant that when the CIPD wished to develop a new certificate course to equip its members with the skills to deliver online and blended training, it was interested in doing so in partnership with The Sheffield College, since it was clear that the college had the expertise to develop a course that would conform to the CIPD's exacting standards. The partners agreed that the principles of LeTTOL were also the principles on which the new Certificate in Online Learning (COL) should be based.

COL was, therefore, based on a series of key pedagogic principles inherited from the LeTTOL course. One of these principles is that those involved in the management, administration, and delivery of the course should have specialist expertise in delivering online learning. In addition, those teaching should be subject experts—it is not enough to have online facilitation skills, though these are of course essential; it is also necessary for an online instructor to have a complete, deep knowledge of the subject at hand. In this case, instructors needed thorough knowledge of the issues, potentials, and challenges of delivering online or blended corporate training.

Learning using online technologies has the potential to remove or at least mitigate the isolation of the learning experience that has in the past been a feature of distance learning. It is this aspect of distance learning that has historically led to demotivation and dropout. Online learning provides the technology for speedy and effective one-to-many communications. Thus, a further key principle was to provide a learning infrastructure that would ensure not only social interactions between learners but, more important, the opportunity to work collaboratively and cooperatively on a series of tasks. Learners on COL are organized into groups, or learning sets, of five to seven people who work together through the course.

In order for online delivery to be effective, instructors must have the ability to provide quick, constructive feedback on any tasks completed by learners. Feedback is likely to be entirely in text, and it is therefore necessary for the online instructor to be able to communicate in writing with great clarity and tact. Internet technologies are fast, and this too must be exploited.

Parallel in importance to the above pedagogical principles is the technological principle of accessibility. All the technologies employed for the delivery of COL should be fully accessible and usable via an ordinary browser (any kind, any version, any platform), and the program designers should be mindful of the fact that many learners will be accessing materials over a dial-up connection and that content should be as quick to download as possible.

Although LeTTOL was the inspiration for COL, and the content and delivery of LeTTOL influenced the design and management of COL, several key differences between the two programs exist:

• The content, links, and assessment activities are designed to reflect the needs and concerns of corporate training.
• To enable a one-day response time from teachers delivering COL, each group would be allocated two instructors, a lead instructor and a support instructor. The lead instructor manages the learning process and gives feedback to individuals and learning sets on the tasks they complete. The support instructor organizes additional discussion activities that are relevant to the course participants but that are not necessarily essential in order to gain the qualification. The support instructor may take over the role of the lead instructor on a temporary basis if the latter is unavailable.
• Senior industry expertise is available to the COL course in two significant ways. One is through the advisory board, which meets twice per year to receive course reports from the course director; to advise on the latest trends and developments in online learning and ensure that the certificate program fully incorporates and reflects them; to act as an impartial sounding board; and to ensure that content and delivery are of the highest standard. The advisory board has members representing the corporate and public sector and the trade union movement. In addition, each COL group is joined for a period of two to three weeks by an industry expert. This person participates in a conference specially set up in order to answer questions and lead discussions relevant to the industry.

A. Pre-course Stage

Traditionally, distance education has not been too successful in terms of completion rates; for a variety of reasons, learners have a tendency to give up before completing distance courses. In addition, for most people even these days, online learning is a new experience. To maximize the likelihood of the applicant completing the course and receiving the qualification, it is vitally important that there be a good match between program and applicant. For this reason, the pre-course phase is critical.

Part of this pre-course phase needs to be simply an information-giving exercise, and detailed information is available about the course both on the web and in print format. However, looking at information is not enough, and the applicant needs to think about exactly what the demands of an online course are likely to be. For this reason, each applicant is required to complete an online pre-course assessment that determines a person's understanding of what the COL course aims to do and of the skills, equipment, and support that each student will need. The pre-course assessment requires applicants to consider these factors and commit to COL's requisites, in particular the management of time and workload in order to meet deadlines and the willingness to participate in collaborative learning. At the pre-course assessment stage, applicants also have the opportunity to contact the course director (via phone or email) to discuss any aspect of the course.

Once completed, the form gets sent to the course director, who reads it and issues one of two possible responses. One is that the applicant is offered a place on the course; in this case, a standard email is sent informing the applicant of this decision and offering some advice and information aimed to ensure that the applicant is successful once the course starts. If the course director is uncertain about any aspect of the pre-course assessment, she makes contact with the applicant to discuss her concerns. In most cases, this leads to a dialogue between applicant and course director that enables the applicant to reach a better understanding of the course demands. This discussion may make the applicant decide to continue with the course or to withdraw or postpone joining; the important thing is that this decision should rest with the applicant. In theory, it is possible that a person could be turned down by the course director; in practice, this has never happened.

B. COL's Aims

COL's main learning objectives are to teach participants to

• be competent in the use of technology necessary for successful online teaching;
• understand the necessity of and demonstrate good organizational skills in using information and communication technologies (ICT);
• understand issues relevant to online collaborative work and group work and be able to facilitate such work;
• recognize how generic teaching skills translate to an online environment;
• identify and apply the skills needed to assist individual learners;
• understand the principles of designing activities and courses for online delivery;
• deploy available technology creatively and appropriately; and
• identify the impact that society may have on online learning.

C. COL's Structure

The course runs four times per year, with starts in February, June, September, and November. The course duration is approximately twenty-three weeks, and learners are expected to study for six to seven hours per week for the whole of this time. Learners go through the course in groups of fifteen to twenty, with each group allocated two instructors. Because of the extreme importance placed upon the learners working in collaborative learning sets throughout the course, learners are expected to meet deadlines in order to work through the course at more or less the same rate. During the course, learners are expected to complete approximately thirty assessed activities, about half of which are individual activities and the other half collaborative. In addition, learners are expected to circulate almost every individual activity to their learning set colleagues.

The course is divided into four units, as follows:

Table 2. eCOL Course Structure

D. Instruction

The quality of management and instruction is essential to the success of the COL course. The following staff support and deliver the course:

• The course director (who is the online learning manager for The Sheffield College)
• The course administrator, an online learning specialist
• A clerical officer
• A team of ten instructors with deep understanding and extensive experience of both the corporate training world and online delivery

All of the above, apart from the clerical officer, have the LeTTOL qualification.

We require COL instructors to have the following skills, knowledge, and commitment:

Technical skills:

• Proficient in the use of word processors, spreadsheets, email, conferencing, and the web
• Capable of providing basic technical advice to their learners

Team skills:

• A willingness to participate in the instructor team by giving support and advice as appropriate
• An openness when participating in communications with other instructors and an ability to be challenging in the positive sense
• Frequent (probably daily) contact between the lead and support instructors to ensure consistency
• Attendance at instructor and moderation meetings

Learner support skills:

• Committed to a collaborative approach
• Able to be diplomatic and friendly, though tough when necessary
• Able to provide encouragement and support
• Intellectual and communication skills to give detailed and constructive feedback on course activities
• Capable of managing the group processes in an influential and discreet way
• A role model of good practice

Updating knowledge and skills:

• Willingness to update and improve their technical skills
• Awareness of existing and developing technologies and their potential for enhancing the learning experience
• Ongoing familiarity with the issues regarding the deployment of online learning in the corporate world

Administrative skills:

• Excellent time and information skills
• Inform the course director of any significant problems that may affect the delivery of the course
• Maintain the security and back-up of their systems and data
• Observe all legal and institutional requirements—for example, the Data Protection Act

E. Quality Control of Instruction

Potential COL instructors are required to go through a selection process that ensures their suitability for the role. The process requires them to complete an application (print or online), participate in an interview (face-to-face or online), and give references.

At the end of each COL delivery, participants complete an evaluation that asks specifically about the quality of instruction. All learners are provided with the contact details of the course director should they wish to raise any issue about the instruction they are receiving. Also, a Feedback Button on the Communications Page of the course content sends mail directly to the course director.

F. Support for Instructors

Online teachers are given an instructor guide containing detailed information about how to deliver and assess the COL course as well as about how to conduct the moderation process. The course director is available to answer questions from instructors either via email or telephone, and an instructor-support WebBoard is available where any teacher can ask for information and advice and receive a range of supportive answers from the instructor team.

During moderation (see Moderation section below), new instructors are put into pairs or trios with experienced ones to ensure that new instructors are assisted through their first moderation. The course director oversees the moderation process.

G. Continuous Improvement: Unit Reviews and Final Evaluation

Online learning is very much a new field of endeavor, which is why it is an exciting area to be involved in, and the pedagogies and technologies are continually moving forward. It is therefore essential to the success of the COL course that a system be in place that systematically evaluates, evolves, and improves the course. Various mechanisms are used to ensure that this is done:

• At the end of each of the first three units, every learner has to complete an online form reflecting on the success of the activities in assisting that learner in reaching the aims of the unit. This is mailed to the course instructor.
• All teachers and the course leader participate in the COL instructor-support WebBoard. Any teacher can suggest changes and improvements to the course—for example, a useful link, an additional paragraph, or the rewording or replacement of an assessment activity. Suggestions are discussed by the course team and, if agreed, implemented by the course leader. These suggestions frequently originate from comments made at the unit end reviews.
• Throughout the course, participants are encouraged to let the instructor and/or the course leader know what they think about the course, including any areas for improvement or clarification.
• Once the course ends, every learner is sent a course evaluation to complete. The course evaluation covers various aspects of both content and delivery. It is confidential and is sent directly to the CIPD.
• Finally, the industry experts on the advisory board give information and advice about improvements that can be incorporated into course delivery and design.

H. Accreditation

Accreditation is key to assuring quality. COL is accredited by the National Open College Network (NOCN), whose website can be found at http://www.nocn.org.uk/. As stated previously, with NOCN accreditation it is essential to express units of learning as learning outcomes, with explicit assessment criteria that the learner is required to provide evidence of having achieved (see Appendix 1). In the case of COL, students must prepare and submit a portfolio of evidence to demonstrate achievement of all the learning outcomes. Course developers created an electronic portfolio structure that enables learners to enter a hot link connecting each assessment criterion with evidence provided; this structure can easily be transmitted as an email attachment, which is often zipped. Some candidates, albeit a minority, prefer to mount their portfolios on the web.

I. Moderation

To quality-assure its accreditation, NOCN appoints an external moderator to visit and report on provision, including sampling the assessment procedure. In the case of COL, NOCN evaluators decided to “e-interview” all students in the first cohort to rigorously assess the new program and its value to the new target audience of trainers (see Appendix 2 for interview questionnaire). The table below provides a summary of the responses to each question received from 29 learners.

Table 3. Questionnaire Results for First COL Cohort

Using the following classification—

a) for responses that were judged to be unqualified positive

b) for responses that were judged to be qualified but positive or neutral

c) for responses that were judged to be negative

The moderator made the following observations in his moderator's report:

The quality of the feedback is absolutely outstanding, not least in how positive learners are about the overall quality of the program—93% unqualified satisfaction for questions 1, 4b, and 6. This is an incredibly positive response for a new program, which is operating near the leading edge of the application of new technology to learning, and of “training the trainers.” There was no negative feedback at all. A 97% unqualified satisfaction score for the quality of the online learning materials and activities is incredibly high, and this is even surpassed by a 100% unqualified satisfaction score for the quality of the dialogue with the instructor. The instructors are to be congratulated [2].

The process of moderating portfolios in order to standardize and quality-assure assessment takes place in two stages:

a) Internal Moderation: Instructors exchange portfolios in groups of two or three and assess each other's portfolios. Two portfolios from each instructor group are randomly selected by the external moderator, and any portfolios considered to be borderline are added to this sample. Decisions are recorded on a moderation form (see Appendix 3). Once the portfolios have been internally moderated, they are forwarded, along with the moderation form, to the external moderator.

b) External Moderation : The external moderator is the final adjudicator on assessment decisions and completes the moderation form before completing NOCN assessment documentation for all candidates. The external moderator also completes a moderation report for each moderation.

IV. Conclusion

This article has been an exploration of the process necessary for putting together a successful online course for the corporate market. In our experience, the key factors for success are:

• Bring together a partnership that combines pedagogical skill with an understanding of the corporate market.
• Understand clearly the pedagogical principles that underpin all learning, and recognize the way those principles are applied when delivering online.
• Offer clear, relevant, and up-to-date content.
• Underpin the content with an assessment strategy that ensures participants are engaged in tasks that build their skills and experience pertinent to their own situations.
• Recognize the critical importance of having expert online instructors who can both support learners and act as role models.
• Ensure a cycle of internal continuous review and improvement for the course by soliciting and acting upon instructor and learner feedback.
• Put in place external and independent monitoring—in the case of COL, the NOCN provided national educational expertise, and the advisory board provided industry expertise.

V. References

1. Cowham, T. Living IT Evaluation and Benchmarking Report. Internal Paper, Learning Skills Development Agency, 1998.
2. Cowham, T. External Moderator's Report on CIPD Certificate in Online Learning (COL): Feedback from the 1st Learner Cohort. Internal Paper, The Sheffield College/OCN/CIPD, 2002.

VI. About the Authors

Terry Cowham is the Assistant Director: Student Services with the Open University in the North West. Previously Chief Executive of the Open College of the North West, based at Lancaster University and Vice Principal of Manchester College of Arts and Technology, Cowham is the External Moderator for the LeTTOL & CIPD Certificate in On-Line Learning  programs.

Julia Duggleby has taught in further education for 27 years, and been involved in online learning since 1997.  She is currently the Online Learning Manager for the Sheffield College, a large further education college in the UK, with responsibility for the management of the college's existing online course portfolio and the development of new courses, often in partnership with other organizations.  She also promotes and support blended learning within the college, and teaches online. She wrote How to be an online tutor, Gower Publishing Limited, 2000; and she was a joint winner, with Fred Pickering, of the 1999 National Information and Learning Technology Association (NILTA) Multimedia Award. She won the 2002 LSDA Practitioner Award for Technology and Innovation in Education—Quality in Staff Development for Information and Learning Technology. And she is Secretary for the European Net-Trainers Association. 

VII. Appendix 1: Learning Outcomes & Assessment Criteria

UNIT TITLE: Certificate in Online Learning in Practice

LEVEL: Three CREDIT VALUE: 3 UNIT CODE: GN73QQ012

VIII. Appendix 2: Text of Email sent to COL Learners on 26/11/01

Dear COL learner:

I am the Moderator for the COL program. CIPD uses independent moderators to quality-assure this program. You should have been informed by your instructor that I am going to contact you. I would be grateful if you could answer the questions below and email them to me. I can assure you that your responses will be completely confidential but will be used to constructively evaluate the program and seek improvements. Of course, should you choose to, you are completely at liberty to copy your instructor in your reply. I would like to take this opportunity of thanking you in anticipation for your help and to wish the very best in completing COL and your future professional development. The questions are listed below and are also attached as an RTF document.

a) Do you feel you are achieving what you set out to achieve in taking part in the program?

b) How do you find the online materials and activities?

c) Please use the headings below to comment on the quality of

Learning materials:

Links to WWW resources:

Dialogue with instructor:

Dialogue with peer learners:

d) How have you found the organization of the program:

Pre-course?

During the course itself?

e) How have you found the approach to assessment?

f) Overall, how have you found the experience of learning online?

g) Will you be able to use what you learned in:

Training?

Enrichment of your professional role?

Online teaching or training?

Program design?

Regards,

Terry Cowham

Assistant Director: Student Services

Open University in the NorthWest

IX. Appendix 3: Sample Moderation Form