Blind Scores in a Graduate Test: Conventional Compared with Web-based Outcomes

M. Hosein Fallah, Ph.D.
Executive-in-Residence
Wesley J. Howe School of Technology Management
Stevens Institute of Technology
Hoboken, NJ 07030

Robert Ubell
Director, Web-based Distance Learning
The Graduate School
Stevens Institute of Technology
Hoboken, NJ 07030

I. INTRODUCTION

While the published literature suggests that there is little or no difference in the outcomes of student results when online learning is compared with conventional classroom instruction [1], an opportunity arose in the spring of 2000 to compare two graduate school environments-one in a conventional setting and the other in a web-based "virtual classroom." The two classes were taught by the same instructor who delivered the same content to both groups.

In order to eliminate potential instructor bias, either in favor of his online students or, conversely, in favor of his face-to-face (FTF) students, a procedure was devised to permit the instructor to grade midterm examinations from both classes without the instructor knowing from which class the exams originated. The "blind" study was performed at Stevens Institute of Technology, a technical and business university that provides undergraduate and graduate education to approximately 4,000 students at its Hoboken, NJ main campus and at some 30 other sites in New Jersey and six other states.

II. GENERAL BACKGROUND

In addition to offering graduate certificates in conventional settings, beginning in 2000, Stevens introduced three entirely online courses as part of an initiative to offer certificates to graduate students over the web. To differentiate its entirely online offerings from those delivered conventionally, the school created an Internet learning environment at its WebCampus.Stevens site, http://www.webcampus.stevens.edu. By 2001, some 25 entirely online courses will be offered.

WebCampus instructors and students employ the courseware application platform, WebCT, with standard tools for interactivity, study and other online learning functions. The site also provides access to the school's digital library, bookstore, advising, application forms, tuition payment, enrollment, and other services online. Admittance to the online graduate school is the same as that for conventional study. Requirements, student background, tuition and other elements are exactly the same for both modes.

The online learning environment, http://WebCampus.Stevens, grew out of Stevens' earlier and ongoing introduction of web-based and other technologically enhanced course delivery. "Electronic Classrooms," in which web-based material supplements FTF classes, is now available in 55 courses at Stevens. In addition, the school delivers graduate programs to corporate sites via interactive video facilities, now in another 25 classrooms.  Together with WebCampus entirely online delivery, the school will offer more than 100 electronically enhanced distance learning courses in 2001.

III. STUDY BACKGROUND

One of the core courses in the Telecommunications Management Graduate Program at Stevens is "Regulation and Policy in the Telecommunications Industry." The course provides students with an understanding of the evolution of telecommunications policy in US, the break up of the Bell System, Communications Act of 1996, and current issues in deregulation of the industry.  Students enroll as part of a requirement for a graduate certificate or master's degree in Telecommunications Management. Parallel sessions are offered on and off campus each semester. 

In early 2000, WebCampus.Stevens began offering the school's Telecommunications Management graduate certificate program online. The regulation and policy course was one of the first to be delivered entirely on the web. Although content remained the same as in the conventional offering, it was redesigned for delivery on the courseware applications platform, WebCT. The online course was offered in parallel with conventional sessions.

At midterm, we decided to compare the effect of web-based learning with conventional education. Even though there has been a good deal of research comparing the effectiveness of online teaching with that of conventional in-class lectures, this study introduced a "blind" test, making it especially unique. 

These are the elements contained in the research environment:

1. Course Content.  Course content, textbook used [2], reading and homework assignments, and quizzes were the same for both classes.
2. Instruction. The instructor was the same in both classes.
3. Examination. The midterm examination given to students in both groups was the same.
4. Venue and Proctoring. Both online and on-campus students were required to come to campus for their midterm exam. Online students had not seen each other or their instructor prior to the day of the test. A teaching assistant proctored the test for both groups.
5. Blind Study Procedure. The proctor shuffled test papers so that they were completely intermingled. A number was assigned to each test and the student names removed.  Following the procedure, the tests were given to the instructor for grading. After grading, student names were matched to tests to record results.

IV. FINDINGS

The test results for each group are shown in Figure 1.  The WebCampus class had seven students whose test grades ranged from 57 to 94 (out of 100 points).  The on-campus class, with 12 students, received midterm grades ranging from 35 to 87. 

Figure 1. Profile of Midterm Exam Grades

From a quick inspection of the raw data, it is not immediately clear which class did better. The distribution of the grades in Figure 2, however, shows that while the means of two samples are close, the on-campus class has a bimodal distribution with some students scoring high and some low.  We will discuss this difference in our observation.  

Figure 2. Distribution of Midterm Exam Grades

We then took a closer look by examining the statistics.  This data is shown in Table 1.  The average score for the online class was 5 points (5%) higher than for the on campus class. The online class scores also had less variation.  The standard deviation for the online class was 13.9, compared to 16.5 for the on-campus class.  To see if the difference in the means is significant, given the small sample size, we used a "t" test. The null hypothesis (the means are equal) could not be rejected.

Table 1. Statistical Comparison of Data from Midterm Exams.

While the study closely follows many results reported in the literature,  a number of observations about the student population in this study are important:

1. Independent Learning.  We believe that online learning placed greater pressure on students to perform independent learning.
2. Student Retention. The number of dropouts from the online class was greater than in the conventional class (5 vs. 2). We believe that students who could not meet the demands of their online class dropped out after a few weeks, leaving a more homogeneous group in the class. This observation is supported by tighter distribution of grades for this group and its smaller standard deviation.
3. Student Population. Even though students self-selected their enrollment in classes offered, all students enrolled in the online class were working professionals. Those enrolled in the conventional class were full-time students and working professionals.
4. Student Personality Characteristics. The students who continued in the online class are more mature personally and professionally, some holding managerial positions in their companies.    

V. CONCLUSION

While this "blind" study confirms results from previous research showing largely that there is little or no difference in student outcomes when online learning is compared with on-campus classroom experiences, the investigators believe that other factors may be significant; namely, it may take greater student maturity to sustain a commitment to self-motivated study in an online environment. 

REFERENCES

1. Russell, Thomas E., The No Significant Difference Phenomenon, North Carolina University, 1999. Available: http://nova.teleeducation.nb.ca/nosignificantdifference/

2. Brock, Gerald W., Telecommunication Policy for the Information Age: From Monopoly to Competition, Harvard University Press, 1998. 

ABOUT THE AUTHORS 

Dr. M. Hosein Fallah is an Executive-in-Residence in the Wesley J. Howe School of Technology Management at Stevens Institute of Technology teaching graduate courses in Telecommunications Management and Technology Management programs.  He has also been teaching course online as part of Stevens Webcampus program. Prior to joining Stevens, Dr. Fallah was Director of Network Planning and Systems Engineering at Lucent Technologies Bell Labs. He has over 25 years of experience in the areas of systems engineering, product/service realization, software engineering, project management, business process reengineering, TQM and R&D effectiveness.  Dr. Fallah is a member of IEEE, ASQC and INFORMS.  He has been an active participant in a number of industry consortia on technology management, R&D best practices and has published and contributed to numerous books and articles on these subjects.

Robert Ubell is Director, Web-based Distance Learning at Stevens Institute of Technology, where he launched the school's online graduate program, WebCampus.Stevens. Ubell negotiated the first e-learning co-sponsorship agreements with the Institute of Electrical and Electronics Engineers (IEEE) and American Association of Mechanical Engineers (ASME). Earlier, Ubell was Editor-in-Chief of Plenum Publishing Corporation, Editor of The Sciences, American Publisher of Nature, and Founding Publisher of Nature Biotechnology. He was also President of BioMedNet and Executive Vice President at Marcel Dekker. Ubell headed his own consulting firm, Robert Ubell Associates. He is the author or editor of numerous works, including more than 40 scholarly articles. He is also a distance learning observer for the Middle States Commission on Higher Education.