A COST-EFFECTIVE MODEL FOR TEACHING ELEMENTARY STATISTICS WITH IMPROVED STUDENT PERFORMANCE

Download PDF version:

Jill L. Lane
jlane@psu.edu
Research Associate/Program Manager, Schreyer Institute for Teaching Excellence
Penn State University
301 Rider Building II
University Park, PA 16802
814-865-9785

John T. Harwood
jth@psu.edu
Senior Director, Teaching and Learning with Technology
Penn State University
229B Computer Building
University Park, PA 16802
814-865-4764

ABSTRACT
Dissatisfaction with teaching a high enrollment introductory statistics course led to efforts to restructure the course to remedy the perceived problems, including lack of student participation, an excessive drain on departmental resources, failure to take into account wide differences in student learning styles, an inability of students to apply statistics after the course, and negative attitudes of students. A cost-effective redesign of the course was implemented that incorporates a learning environment that is student-oriented, involves active student participation and hands-on experience with data analysis, uses technology to reduce costs through labor-saving techniques including low-stakes computerized testing, and sharing of resources enabled by a web site for course management and delivery of course materials. Responsibility for learning basic concepts was transferred to students and motivated by readiness assessment quizzes. The redesign led to about $125,000 in cost savings to the department.

KEY WORDS
Large Classes, Uses of technology, Readiness Assessment, Cost Effectiveness

I. INTRODUCTION

At the main Penn State University campus, the basic elementary statistics course is taught in the fall, spring, and summer semesters with an annual enrollment exceeding 2,000 students. An additional 400 students take statistics each year at the university's twenty other campuses. It is a pre-calculus, introductory survey course with students primarily from the social and behavioral sciences. It was redesigned and implemented in the fall semester, 2000.

In the (previous) traditional format, students attended three lectures and two recitation meetings per week. Four faculty members, instructors, and/or senior graduate teaching assistants (GTAs) lectured to about 240 students in each of four classes. Twelve graduate teaching assistants each handled two, one-hour recitation sections of about 40 students. GTAs also held office hours and graded exams. This structure (which is the norm at most large universities) was labor-intensive, requiring a substantial amount of faculty and GTA time per semester, and it created resource problems for the department. More importantly, the traditional structure was not as effective academically as it should have been.

Four years ago, a grant from the Pew Foundation’s Center for Academic Transformation was awarded to the university to redesign the course. Three groups cooperated in the effort: The Schreyer Institute for Teaching Excellence, Educational Technology Services, and the Department of Statistics. A primary criterion in evaluating awards specified that redesigned courses should either enhance the quality of instruction or reduce costs or both. We had several concerns about the previous course:

1. The traditional format did not address the broad range of differences in student learning styles and quantitative skills. Students with weak skills need more individual attention and more opportunity for group collaboration, while students with strong skills would benefit from having more opportunity to explore the material more fully.
2. The original format did not encourage active participation. It was difficult for students to ask questions, discuss the material, or collaborate with other students. There was not enough hands-on experience with data analysis and collection.
3. The traditional structure required twelve GTAs (four per large class) each semester. It was difficult for the department to identify, much less allot, this many qualified assistants for the course. Most graduate students in statistics have undergraduate degrees in mathematics or a scientific discipline, limiting the effectiveness of the statistics instruction they can provide.
4. Students were unable to apply statistics in follow-up courses.
5. They had a negative attitude towards statistics.
6. Retention of subject matter appeared to be very short.
7. As initially designed, the course did not provide sufficient tutoring assistance for students. GTAs working with a particular instructor had only two or three office hours for students within their own recitation section, limiting accessibility of students for individual attention overall.

A. Philosophy and Goals for the Redesigned Course

We made several learning assumptions—for example, learning is enhanced by collaborative group activities; with supporting resource materials, students could learn many of the course concepts on their own. We did a lot of reading about learning groups [1], [3], and [4]. We looked at technologies and assumed that (a) computer labs dedicated to instructional use would be available, (b) statistical software would be available online in the labs, and (c) instructors and students can communicate synchronously and asynchronously. Finally, we adopted some guiding principles:

• Give responsibility for learning basic concepts to students and let the instructor's role be that of a facilitator
• Provide as much hands-on practice as feasible
• Use technology appropriately
• Reduce time between concepts and applications to a minimum

We attempted to free ourselves of preconceived notions about course content and sequencing. We then constructed a new course that we felt met our course goals. In general, the goals were to:

• Provide a learning environment that is student-oriented rather than teacher-oriented
• Incorporate more active student participation and frequent hands-on experience with data analysis and interpretation of concepts
• Use computers for testing, web-based courseware, statistical analysis, and resource-rich course web sites
• Provide timely feedback to students in their problem-solving tasks, analysis, and interpretation
• Increase opportunities for collaboration with other students and with the instructors
• Incorporate group work in various ways, including low-stakes quizzes and projects

B. The Redesigned Course: Overview

Each of the three participating units involved in the restructuring made key contributions to the effort. Consultants at the Schreyer Institute for Teaching Excellence recommended the revised course structure that was eventually adopted. Educational Technology Services created a web-rich course management system [6] that has been highly effective and well received by the students. A group of faculty, instructors and TAs, with the advice and assistance of these two units, developed the course materials to be used. The outcome is a restructured course that both cut costs of instruction substantially and results in enhanced student performance, thereby fulfilling the desired objectives of the Pew Foundation’s Center for Academic Transformation.

The redesigned course has one large group meeting (LGM) per week per class of 240-320 students. The traditional recitation sections were changed to computer-mediated workshops (labs) that include technology-based independent learning materials and computerized testing to give students more practice time and feedback. Instructional roles were shifted from information presentation to learning facilitation. The combination of smaller classes and computer-mediated data workshops enables faculty to have more one-to-one contact with individual students. Faculty members are able to address the different needs of individuals, and students can be challenged according to their own skill levels. Computer-based classes enable students to work in teams, which generates participation that is more active. There is frequent hands-on experience with statistical analysis and with the visualization of concepts. GTA roles shifted under the new structure from instruction to guidance. Technology-based instruction and collaborative activities enable students to explore material that was formerly taught by teaching assistants. Faculty members guide many of the computer lab meetings (formerly called recitation sections) that in the old format were led by teaching assistants. GTAs are paired with faculty and an undergraduate intern in the labs, enabling the faculty member to model ways to facilitate learning. Student understanding of the course material is reinforced in the computer labs through individual and group work on activities designed specifically for this purpose. Students are told at the beginning of the course that a major portion of responsibility for attaining the course goals is being assigned to them, that is, they are expected to learn many of the concepts on their own through assigned reading and studying, and that they will be motivated to do so through frequent Individual and Group Readiness Quizzes (described below). They are also told that (a) lectures will be given in the LGMs on days when there are no Readiness Assessment Quizzes (RAQs), (b) in the computer labs they will be working on 'activities' in pairs or small groups to apply what was learned in the readings, (c) lab quizzes consisting of 5-8 questions will also be given in 15-20 of these labs. Items on the Lab Quiz will be based on the activity for the lab and general concepts being illustrated by the activity, and (d) some classes and labs will be reserved for work involving the integration of course content, such as evaluating scientific articles and completing group projects.

As a result of the restructuring, the most important learning goals for the course now require students to:

• Actively engage with course materials and other students
• Actively participate in data analysis and design
• Understand the reasoning by which findings from sample data can be extended to larger, more general populations
• Understand and apply basic concepts of statistics (e.g., variables, inferences, probability, types of designs, etc.)
• Design, conduct, and analyze a scientific research study, including using computer software
• Critically evaluate the results of scientific studies

As redesigned, the course consists of two types of classes: Computer Labs and Large Group Meetings (LGMs). The two computer labs and one LGM each week are sequenced as Lab>LGM>Lab or as LGM>Lab>Lab, depending on the time the class is offered and availability of computer labs.

In summary, the following changes were implemented in the redesigned course:

• Reduced weekly lectures from three to one
• Changed recitations to technology-based classroom meetings
• Offered interactive, Web-based course materials and computerized testing
• Offered collaborative and hands-on learning opportunities for students
• Greatly reduced GTA time grading exams
• Shifted faculty and GTA roles from instruction to learning facilitation, and
• Provided additional help through a tutorial lab staffed by instructors, GTAs, and undergraduate interns

Faculty class contact time has remained at three periods per week, but time devoted to preparing for classes and preparing quizzes and exams has been reduced considerably by eliminating duplication of effort.

The reasonably sophisticated but easily navigated Web site contains not only the management aspects of the course but also a large number of student aids and resources (solutions to problems, study guides, supplemental reading materials for topics not otherwise treated in the text, self-assessment activities, and more). Having assignments, quizzes, exams, and RAQs on the community web site saves a considerable amount of instructional time, for both instructors and TAs.

1. Readiness Assessment Quizzes

With just one LGM each week, the amount of time available for formal lecturing is minimal. There are about six or seven lectures in LGMs in the 15-week semester. There is some lecturing in labs, up to 10-15 minutes maximum. Students are given the responsibility for their own learning and are given weekly reading and homework assignments, and to 'motivate' them to do it in a timely fashion they are given Readiness Assessment Quizzes (RAQs).

RAQs have two major components, and sometimes a third, depending on an instructor's use of them:

• An individual component
• A group component, and
• An appeal process

RAQs were developed by Michaelson [2] as an instructional and assessment tool. Students are given reading assignments before classes and before instruction on the material. The goal of these reading assignments is to provide motivation for students to learn some of the basic concepts of the course on their own. After the reading assignments students come to class and take a RAQ, made up of true/false and multiple-choice questions. These questions test knowledge and understanding of general concepts and principles rather than small, detailed facts. The goal of the individual RAQ is to ensure accountability and understanding of the reading assignments. Students take the individual RAQ first, turn it in, and then immediately re-take the same test as a group (previously set up) of three to five. The goal of the group RAQ is to foster students helping one another to comprehend ideas that they may not have gotten on their own. If the instructor chooses to do so, students are allowed to appeal any incorrect answers based on the quality of the question or a justification for the answer choice. Each student receives an individual and group grade for each RAQ.

The instructor uses the feedback from the individual and group RAQ scores to determine where students still have misconceptions or misunderstandings. The concepts that students did not get on their own can be used to guide and inform instruction. The feedback helps the instructor focus instruction and activities on application of the course content rather than spending time covering the concepts that students can easily obtain through self-directed reading and learning. Course activities are typically completed in pairs or groups. The RAQs and the content covered on them are used as a means to prepare students for the application of the content in problem-based activities. As for the individual and group collaborative activities, we were pleasantly surprised at the students' positive reactions to not being lectured to and instead being able to work in groups in the labs to apply what they had learned from the resource.

RAQs cover 'natural units' or modules, usually one to three chapters in the text. Each semester about five or six RAQs are given. RAQs provide a powerful motivator for students to read material prior to classes (since it is a major component of their grade) and to keep up with work on a regular basis rather than trying to study at the last minute before an exam. The assessment of student understanding of concepts using RAQs has proven to be very effective in detecting areas in which students are not grasping the concepts, thereby enabling corrective actions to be taken in a timely manner, and in preparing students for higher level activities in the computer labs than previously. As a result, students have been helped in building skills, as the evidence of pre- and post-tests show.

Student perception of the importance of RAQs is evident in the results from Innovation and Quality (IQ) survey data where the majority of students (55%) rated the RAQs as one of the most important aspects of the class. Seventy-five percent of respondents believed that periodic RAQs help them keep up with the readings and that they were vital for their learning and understanding of the content. As voiced in focus groups, students felt that the RAQs helped by promoting recognition of gaps in their understanding. In addition, students liked the opportunity to work in groups and interact with others in the class. Most students emphatically suggested keeping the Readiness Assessment Testing as part of the course.

2. The Redesigned Course: Details

Broadly speaking, there were four major components of the redesigned course:

A. Course Materials
B. Uses of Technology/Computer Aspects
C. Assessments of Student Learning
D. Tutorial Sessions

A. Course Materials. As an aid to facilitate student mastery of concepts, we developed the following materials for their use:

• Study Guides. The study guides for each chapter of the text, each about two pages in length, pointed out to students the important points they should focus on in their reading and studying of course concepts.
• Computer Lab Activities. About 70 individual and group activities were created for students to work on in the computer labs. The individual activities were structured to reinforce concepts as ‘practice,’ while the group activities were less structured and more challenging and to be done by groups of four or five students.
• A Guide for Using the Minitab Statistical Software Package. To facilitate use of the Minitab Statistical Software used in the labs, a compact guide for performing statistical analyses was prepared for students to use.
• A Summary Table of Statistical Procedures. The two branches of statistical inference are estimation and testing. We assembled a summary table designed to enable students to identify and select the appropriate technique to use among the statistical inferences we covered in the course and direct them to the appropriate Minitab commands to carry out analyses.

B. Uses of Technology/Computer Aspects. We use computers and other technology for the following:

• Web Course Management System [6]. An excellent web site was designed by Educational Technology Services that has the following features and components:
  • An Agenda for each class meeting, specifying the content, activity, and any special news items students need to be aware of
  • Reading Assignments
  • Homework Assignments and solutions
  • All course materials listed in A (above)
  • Links to other web sites (e.g., to the Gallup site for polling)
• TestPilot [5] (recently purchased by McGraw-Hill). A feature-rich tool that we use for:
  • Low Stakes Computer Lab Quizzes. In the last 5-10 minutes in the computer lab, students take a short lab quiz online using TestPilot. There are usually seven or eight multiple choice items on the quiz and the students may consult with a lab partner or all of their group members (three or four per group) in answering the questions. The purpose of this, of course, is to encourage 'students teaching students'. Student responses on the lab quizzes are sent directly to a file, the results are summarized and made available to instructors to assess student understanding of the concepts covered in the lab. At the next class meeting instructors review any concepts that students did not grasp well.
  • Student Surveys on attitude and course attributes
  • Data Collection. We collect two sets of data from students each semester, one for use in statistical analyses and examples and the second for a major group project. Students participate in the design and creation of both data surveys. The benefits of this are that students have a sense of ownership of the data, gain a greater understanding of survey techniques and construction, and have data on issues for they can relate.
• Statistical Software. As noted above, we use the Minitab Statistical Computing Package in the computer labs. There are 82 PC’s in the computer lab, so each student has one available for use. However, since students can work individually, in pairs, or in groups, not all PC’s may be in use at times.

C. Assessments of Student Learning. We have built-in a mix of assessment instruments for assessing student performance, including the following (percentage of student grade):

• 15-20 online low states quizzes, using TestPilot (15%). These are given at the end of labs.
• Five or six Readiness Assessment Quizzes (24%). These are given in the Large Group Meetings (LGMs) or in the labs, depending on various factors. The Individual RAQs consist of 13-18 multiple-choice questions. The Group RAQ is on the same set of questions and is given immediately after the Individual RAQ. About half of the items are on previously discussed topics and the other half on new material not previously covered.
• Two-in-class examinations (16%). These in-class exams are open-ended questions (that is, show your work) counting for about one-third of the grade and multiple-choice items about two-thirds. GTAs grade the open-ended questions and the multiple-choice items are machine-graded—answers are placed on bubble sheets (Scantron Forms). The multiple choice portion is graded immediately (usually same day as the exam), scores emailed to students, and an item analysis performed showing (i) student responses, (ii) % correct for each item, (iii) a test validity measure, and (iv) average and standard deviation for all sections of the class. Instructors use the item analysis as an assessment instrument for identifying concepts that students did not grasp well.
• Group projects (14%). Students are given two substantive projects, the first about the fourth week of the semester and the second during the last two weeks of the course. The first is moderately well structured and the second is unstructured, with just general guidelines. As noted above, a survey is developed specifically for the second project, cooperatively between instructors and students. Students are given a scenario, like 'A President of a music company has experienced a downturn in his business. He asked the marketing department to collect data on aspects related to music, and the Director of Marketing contacted Penn State to do a survey of college students for them. Naturally, the Department of Statistics was contacted to do this.’ Therefore, with the help of the students a survey is created. As a group project, students are asked to analyze the data, interpret it, and prepare a report for the Director of Marketing at the company. The requirements imposed on the project specify that students are to perform at least six different statistical techniques in their analysis (e.g., compare two means, two proportions, regression, chi-square test, analysis of variance, and so on).
• 12-13 weekly homework assignments. These are graded by undergraduate interns under the supervision of GTAs. (7%)
• Special Homework Assignment. This assignment contains 20 problem statements, in which students must specify a null hypothesis and the statistical technique to use, chosen from a set of 10 (4%)
• Final examination (20%). The final exam is usually entirely multiple-choice items because of a requirement to assemble and report grades to the registrar's office in a very short time period (two days or less). If the exam is scheduled early in the finals week, a portion of the exam (up to 50%) may be open-ended.

D. Tutorial Sessions. In addition to office hours, each week there are at least three 'tutorial' sessions that are set up to assist students having difficulties with the concepts and/or with homework. They are also used to answer questions about the reading assignments prior to the RAQs. Tutorials are not designed as 'lectures' or to solve the assigned homework problems, but rather to give students feedback on the material. They are conducted by the GTAs.

2. The Redesigned Course: Details

a. Student Performance

To determine the impact of our revised course on student learning, a content knowledge test consisting of 18 items was developed prior to the restructuring of the course. It was administered at the beginning and end of the spring 2000, fall 2000 and spring 2001 semesters. During the spring 2000, two sections were taught in the traditional format (n=340) while one section was taught as a pilot using the revised approach. In the fall and spring semesters of 2000/01, all classes were taught using the new format. A 20-item test on 'choosing the appropriate statistical technique from a set of 10 was created and has been used as part of final exams every semester since then. Statistics on the number of students who received D's, F's, student grade point averages, and number of course dropouts were compiled for the 5-year period 1996/97 through 2001/02.

The results of all these types of evaluation revealed many positive results. The pilot and redesigned classes outperformed the traditional class on the final test of content mastery by 10% to 13% (60%: traditional class, 66% in the pilot class, 68% in the redesigned classes). The improvement in performance in the redesigned class was greatest on concepts. On technical aspects (working with formulas, for example) the traditional class performed slightly better. Students in the restructured course were able to identify the correct statistical technique to use about 86.5% of the time, about 11% better than the 78% correct rate for students in the traditional course. This is viewed as a consequence of lab work. The percentage of students receiving a D, F, or dropped the course decreased from a rate of about 12% in the traditional course to about 9.8% in the restructured course, or about 18%. The average student Grade Point Average was essentially unchanged: 2.974 in the traditional course and 3.015 in the new course.

b. Cost Effectiveness

As a result of the course restructuring, substantial cost savings and other benefits have occurred:

• The number of teaching assistants assigned to the course has been reduced by 2/3 as a result of the redesign. Previously, one TA was used per 80 students; with the redesigned course, one TA is assigned per 240 students.
• Course enrollments have increased by 20-25% in the past three years following the course restructuring, resulting in four large classes of 300-320 each instead of 240 previously.
• 15-16 undergraduate students (called ‘Interns’) are recruited to serve as assistants in the computer labs and to grade homework assignments. The cost of all of the undergraduate students involved in the course is equivalent to about one TA. This greatly reduces the graduate TA's workload, to the point that the department now assigns each TA three lab sections instead of two.
• The reduced need for TA's provides opportunities for training of future GTAs for teaching responsibilities or to reassign them to research, an unexpected bonus.
• It also means that the department can be more selective in its assignments of GTAs to the course.
• The department has accrued cost savings of approximately $125,000 per year. These savings are retained by the department.

These benefits and cost savings result from the labor saving techniques for both instructors and teaching assistants by employing technology judiciously and by the changes in the redesigned course. Explicitly, they result for the most part from the following:

• TESTPILOT, low-stakes quizzing/data collection software
• Readiness Assessment Quizzing
• Sharing of Resources by instructors and GTAs, to reduce preparation time
• A 'Shared' web site, for course management and delivery of course materials
• Undergraduate interns

II. SUMMARY AND CONCLUSIONS

We have described a course redesign of a highly enrolled elementary course in statistics that combines teaching and learning innovations in higher education with the appropriate use of technology, resulting in improved student performance and substantial cost savings.

III. REFERENCES

1. Felder, R. M. and R. Brent. Navigating the Bumpy Road to Student-Centered Instruction. College Teaching, 44(2): 43-47, 1996
2. Michaelson, L.K. Myths and Methods in Successful Small Group Work. The National Teaching and Learning Forum, 8(6): 1-7, 1999.
3. Michaelson, L.K., Black, R.H., and L.D. Fink. What Every Faculty Developer Needs to Know About Learning Groups. In L. Richlin (Ed), To Improve the Academy: Resources for Faculty, Instructional and Organizational Development, New Forums Press, 1996.
4. Springer, L., Stanne, M.E., and S.S. Donovan. Effects of Small-Group Learning on Undergraduates in Science, Mathematics, Engineering, and Technology: A meta-analysis. Review of Educational Research, 69(1), 21-51, 1999
5. Test Pilot: see http://www.clearlearning.com
6. The Stat 200 website: http://stat200.stat.psu.edu/

IV. ACKNOWLEDGEMENTS

We wish to express our appreciation to the Pew Foundation's Center for Academic Transformation for a grant that made the redesign possible. We are also grateful for our colleagues' long hours of hard work and contributions in carrying out the restructuring. Finally, we are thankful for the university's administrative support of the project.

V. ABOUT THE AUTHORS

Dr. William Harkness is Professor of Statistics at Penn State University. For the past 15 years his main interests have been related to statistical education, particularly with teaching classes with large enrollments. He is currently involved with restructuring three other courses at Penn State, with support from NSF and the Schreyer Institute for Teaching Excellence. He has collaborated extensively on research in statistical education with faculty in the College of Education, particularly in Instructional Systems and Design and Curriculum and Instruction.

Dr. Jill L. Lane is a Research Associate/Program Manager of Course and Curricular Development at Penn State's Schreyer Institute for Teaching Excellence as well as an Affiliate Assistant Professor of Instructional Systems in the Department of Learning and Performance Systems. Her research focuses on teaching and learning innovations in higher education. She has served as project manager for multiple large-scale curriculum reform projects and has consulted and collaborated with faculty on the redesign of courses to include active and collaborative learning. She has conducted workshops for faculty on problem-based learning and active and collaborative learning at Penn State as well as professional conferences.

Dr. John T. Harwood is Senior Director of Teaching and Learning with Technology, a unit of Information Technology Services. The mission of TLT is to help faculty improve teaching and learning through judicious use of technology. He serves as Penn State's representative to the Cable in the Classroom (CIC) Learning Technologies initiative. He also provides a leadership role in several national technology initiatives, particularly the IMS project and Educause. As a teacher and scholar, he holds a joint appointment in the School of Information Sciences and the department of English, and he has published widely on critical issues in both the humanities and technology.