EVALUATING ONLINE DISCUSSIONS: FOUR DIFFERENT FRAMES OF ANALYSIS

Download PDF version:

ABSTRACT

This study uses four different “frames” to analyze 17 online discussions that occurred in two doctoral-level classes in educational leadership. Two of the frames were developmental models: King and Kitchener’s Reflective Judgment Model and Perry’s model of intellectual and ethical development. Two of the frames captured levels of thinking: Garrison’s four-stage critical-thinking model and Bloom’s taxonomy of educational objectives. Of the 278 individual postings, 45.3% were at levels five through seven of the King and Kitchener model, 100% were at levels five through nine of the Perry model, 52.2% were at the two highest levels of the Garrison model, and 54.3% were at levels four through six in Bloom’s taxonomy. These results seem appropriate to the level of response expected of doctoral students. For each frame, the analysis resulted in additional findings. The study concludes that each frame has value and focuses attention on different aspects of the student’s thinking as evidenced in his/her posting to an online discussion; however, some frames are more difficult to use than others, which argues for specific training and/or tailoring the topic of discussions to address issues in a particular manner. Lastly, the question initiating each of the online discussions influenced the level of the responses from students. Each frame has the potential to illumine students’ online discussions, although using multiple frames may have more benefit than using any one frame exclusively.

KEY WORDS

Online learning, evaluation of online discussions, developmental models, critical-thinking, Bloom’s taxonomy

I. INTRODUCTION

A. Purpose

An earlier study by Meyer [1] used Garrison’s [2] four-stage cognitive-processing categories to analyze several online discussions of graduate students in educational leadership classes. It was a useful exercise that helped to analyze what occurred during the discussion and identified ways to improve future discussions. But it also generated an interest in locating and evaluating other means of analyzing online discussions. Were other frameworks or rubrics useful? Would the frame used to evaluate an online discussion affect the analysis and its conclusions? This seems a reasonable assumption, although it needed to be evaluated on real student online discussions. And if this assumption turned out to be a moderately accurate assessment, it would indicate that the choice of frame or tool for evaluating online discussions would need to be carefully made or that multiple assessments should be chosen.

II. LITERATURE

A. Introduction

There has been an increased use of online discussions within courses that are exclusively online or use online technologies to enhance on-campus courses. While no data on the phenomenon exists, it is clear that chatrooms and threaded discussions have entered many higher education courses to provide synchronous or asynchronous (respectively) opportunities for students to interact with faculty and other students on course topics or issues. One of the advantages of online discussions is that it leaves a written, printed record that can be referred to by the student for reflection and information and also analyzed after the discussion by the faculty. Drops [3] has argued that simply counting the number of postings made by students “does not necessarily lead to quality interaction, and quality does not readily come with quantity” (p. 8). The faculty may be looking for evidence of specific learning related to course objectives, or participation of the student in required course activities, or they may need to analyze what happened and how to improve future online discussions. The written record of online discussions is a boon to researchers and faculty who wish to better understand the dynamics of online course work and the thought processes of students.

However, only a few frameworks have been specifically developed for analyzing online discussions. Are there other frameworks developed prior to the boom in online learning that might be useful? These frameworks may not have been used previously to analyze online discussions, but do they add some dimension that is helpful to the researcher or faculty in charge of assessing student learning? The issue is two-fold: can they be applied to online discussions and what do they tell us about student’s thinking online?

At least for the current exercise, rubrics were not considered. A rubric is an “authentic assessment tool which is particularly useful in assessing criteria which are complex and subjective” [4]. Rubrics improve the objectivity and consistency of assessment and require faculty to clarify criteria beforehand and to do so in very specific terms [4]. Two excellent examples of rubrics are Edelstein and Edwards [5], which assesses the effectiveness of student participation in online discussions during an entire course, and Roblyer and Ekhaml [6], which assesses the interactive qualities of distance learning courses.

B. Developmental Frameworks

Two developmental frameworks that have been useful for understanding student development are King and Kitchener [7] and Perry [8]. These two frameworks were chosen for four reasons. First, extensive research literature exists on both models to help others understand their implications, and they seemed to capture different qualities of student thinking. Second, they were developed prior to the boom in online learning, and it was intriguing whether they might be applied usefully in this new medium. Third, it was not clear whether they could be applied to the online world, and fourth, it was not clear that doing so would be helpful to understanding the online conversation. The only solution seemed to try them out and determine what could be learned from the experience.

King and Kitchener [7] developed a seven-stage model of reflective judgment (see Table 1) that captures the development of reasoning skills in students. The process of evaluation is based on presenting complex situations to students and analyzing how students think about ill-structured problems. In suggesting the use of the King and Kitchener [7] model to evaluating online discussions, there are several possible problems. First, the method by which King and Kitchener tested and analyzed a student’s developmental stage required extensive time on the part of the student and tester and may not translate into situations where students make brief comments and are separated by space and time from others in the class. Second, the model depends on the student’s interpretation and evaluation of a situation and not on the students’ response in an online discussion to a course topic. And yet the model is intriguing because its substantial research base appears to capture unique and valuable thinking skills of students, and it represents a new way to evaluate the thinking of students taking part in online discussions. Whether the model can be fruitfully or appropriately applied to analyzing a course-related online discussion is an important question, and one worth asking. In applying this seven-stage developmental framework to online learning, will faculty be able to assess students’ developmental stage from student postings? Can it (and should it) be used in this fashion?

Table 1. Stages of Reflective Judgment: King and Kitchener (1994, p 31)

Perry [8] has developed a framework or model of intellectual and ethical development for college-level students. This nine-stage model (Table 2) tracks student development from basic duality, through multiplicity, relativism, and several stages of commitment (the descriptions in Table 2 are necessarily brief and incomplete). This model has been extensively researched, but has not been applied to online students nor their class-related online discussions. In suggesting the use of the Perry [8] model to evaluating online discussions, there are several possible problems. Will the postings be of sufficient length to classify the student’s contribution? Will the posted questions or topics for discussion elicit the types of responses that can lead to a classification being made? Whether this model can be appropriately applied to analyzing a student through his or her responses in a course-related online discussion is another important question worth pondering. In applying this nine-stage development model to online learning, will faculty be able to assess students’ developmental stage from student postings? And if it can be done, will it result in useful insights into the applicability of the model to online discussions? And finally, can either the King and Kitchener [7] or Perry [8] model be used by average faculty who have not been trained in their use?

Table 2. Developmental Positions: Perry (1999, p. flyleaf)

C. Levels of Thinking

Two examples of approaches to evaluating different levels of thinking include the ideas of Garrison [2] and Bloom’s [9] taxonomy, the cornerstone of many teacher education programs and lesson plans. These two ways of assessing levels of student thinking were chosen for two reasons. First, they were familiar to the author (Garrison had been used in an earlier study [1] and Bloom has been a mainstay of teacher preparation programs for decades). Second, they appeared to capture different approaches to characterizing student thinking. Third, could they be used to assess online discussions and fourth, would the results of doing so be helpful to understanding students’ thinking?

Garrison [2] has developed a four-stage cognitive-processing model that has been used to assess critical-thinking skills in online discussions (see Table 3). Garrison [2] have established indicators and examples to help faculty better classify student responses into a four-stage process: 1) triggering (posing the problem), 2) exploration (search for information), 3) integration (construction of possible solution), and 4) resolution (critical assessment of solution). In research conducted by Garrison [2] and Meyer [1], the incidence of contributions coded as resolution may have been due to the need for more time to reflect on the problem and that individuals would hesitate to offer inadequate solutions in a public setting in order to avoid rejection.

Table 3. Critical Thinking Categories: Garrison et al. (2001, p. 15-16)

As suggested by Drops [3], the level of learning demonstrated in students’ postings to online discussions could be assessed by using Bloom’s taxonomy “to distinguish simple recall from comprehension, analysis, and evaluation.” Bloom’s taxonomy is reproduced in Table 4 [9, 10]. This taxonomy is neither a developmental model for students nor a rubric for assessing online discussions, but a classification of educational objectives used in the creation of lesson plans and educational goals and assessments. But Bloom’s taxonomy was chosen for analysis based on three reasons. First, it is a framework that will be familiar to many faculty who have had rudimentary training in designing learning objectives. Second, it also has an extensive research and theoretical base that can be helpful in grounding results in prior research findings. Third, it could be easy to use for the average faculty. While using Bloom’s taxonomy to evaluate students’ online thinking is a novel use, it may provide another means of evaluating students’ contributions to online discussions and is worthy of analysis.

Table 4: Taxonomy of Educational Objectives: Bloom (1956) and Krumme (n.d.)

Clearly, these may be promising tools for evaluating online discussions. Will each be useful in some way, or will some of these models—developed for an entirely different use—be inappropriate for online discussions? Furthermore, does the frame used to evaluate an online discussion affect the analysis and its conclusions? And if so, how do we choose a frame or tool for evaluating online discussions? Can they be used by an average faculty person who is not trained in the model? In other words, the overriding research question is will these frameworks be useful and usable?

III. METHODOLOGY

Online discussions that occurred in two doctoral-level classes in educational leadership held over two semesters in 2002–03 were the subject of the analyses. Each discussion was printed, dated, and individual student names blacked out and replaced with codes. A total of 17 different online discussions were analyzed, comprising 278 total postings by the same ten students in each class. It was felt that using postings by the same students over an extended period of time would approximate more closely the procedure of extensive interviews used in the King and Kitchener (1994) and Perry (1999) research.

Each discussion was analyzed against the two developmental frameworks (King & Kitchener, 1994; Perry, 1999) and two models of thinking (Garrison et al., 2001; Bloom, 1956 and Krumme, n.d.). For brevity’s sake, these are referred to as the “four frames” throughout the study. If an individual posting within a discussion could not be categorized, it was deemed “not categorized.” If a posting—many of which were quite lengthy—could be categorized at multiple levels, the level or category most consistent with the entire posting was used. This analysis produced a total number of postings for each category of the four frames; these results are displayed in Tables 5 through 8 for each of the four frames. Also, throughout the process, the author noted problems during the coding process or analysis.

A note about the goals of the online discussions is needed. All of the discussions were led by a student who selected the question or topic they wished to discuss with their class peers related to that week’s class reading. The value of the frameworks appeared to be their focus on students’ thinking, not just their knowledge of the topic being discussed (for example, higher education finance, governance models). Thus, the topics of each discussion changed each week and each student could approach his or her leadership of a discussion based on personal interests. This variety helped keep students involved in the online discussions and introduced variety into the discussions.

Three limitations are worth noting. First, the researcher, while having read and studied the developmental frameworks of King and Kitchener [7] and Perry [8], is not trained in using these models for evaluating students. And yet it was one objective of the study to assess the usefulness of these frames because most online educators may not have the funds to hire professionally-trained coders and will need to do coding themselves. Thus, these frames must be evaluated for their usability by amateurs. Second, despite coding the names of the students on the printed online discussions, personalities and points-of-view were known to the researcher. Third, the online discussions were coded by the researcher and instructor of the class.

IV. FINDINGS

A. The Postings by Frame

Tables 5 through 8 present the summary information on how the 278 individual postings were classified by the four frames. Each table is followed by the major findings resulting from the analysis.

Table 5: Number of Postings by Stage of Reflective Judgment: King and Kitchener (1994, p. 31)

At least three observations can be made from the use of the King and Kitchener frame for these online discussions. First, the large number of postings at the third level (30.6% of the total) is the result of several discussions being triggered and conducted by the request for and submission of personal reflections or personal experiences. Second, the second largest number of postings at the sixth level (24.5% of the total) was the result of several discussions that requested other students to provide “pro” and “con” arguments on an issue. Third, 45.3% of the posting are at levels five through seven, which likely is the result of analyzing online discussions conducted by doctoral students and the emphasis in doctoral education on justification and reasoning while responding to questions.

Table 6: Number of Postings by Developmental Positions: Perry (1999, p. flyleaf)

Use of the Perry frame for analyzing the online discussions revealed three important insights. First, this frame was the most difficult to use, resulting in the highest number (22.3%) of postings that could not be categorized. Second, postings of these doctoral students were exclusively at the fifth through ninth levels, which may reflect their age, maturity, and/or the setting for the discussion. Third, consistent with Perry being a classification scheme for individuals and not individual postings to a discussion, two individual students were largely consistent in the level of their contributions, submitting at least 15 postings each at the same Perry level.

Table 7: Number of Postings by Critical Thinking Category: Garrison et al. (2001, p. 15–16)

This frame revealed three insights into the structure of the online discussions. First, each discussion was led by a student who would often ask a leading question of his or her classmates, resulting in 18.3% of the total postings being classified as “triggering questions.” Second, the majority of postings (a total of 59.4%) were focused on exploring and integrating ideas. Third, while only 19.8% of the postings were classified as “solutions,” 40% of these “solution” postings occurred in five discussions that specifically requested students to resolve a problem. In other words, the nature of the triggering question influenced the level of response from the students.

Table 8: Number of Postings by Taxonomy of Educational Objectives: Bloom (1956) and Krumme (n.d.)

Use of this frame revealed three insights into the online discussions. First, almost one-third of the postings were classified as analysis, revealing that most discussions were likely oriented toward understanding and justifying positions. Second, postings are more evenly spread out across the categories, with no one category not being represented. This may mean that in most discussions students need to contribute knowledge, comprehension of information, and application of information, as well as synthesizing and evaluating information (although this last was done less frequently). Third, 40% of the higher-level postings (those classified as synthesis or evaluation) occurred in five discussions that specifically requested students to work on solving the problem presented in the question. In other words, the nature of the question influenced the level of response from the students.

B. Patterns from the Analyses

When the descriptions of postings are viewed from the perspective of a combination of classifications from King and Kitchener, Perry, Garrison, and Bloom, a remarkable finding becomes evident. Of the 278 total postings, 206 postings (or 74% of the total postings) were coded by all four frames. Of these 206 postings, 63% or 130 postings had unique combinations of the four frames; only 37% of these postings were coded by combinations of the four frames (e.g., KK3, P7, G4, B4) that were duplicated by other postings. In other words, what is most compelling about this analysis is its lack of consistency across the four frames. If one were expecting consistent combinations across the four frames (e.g., the third level of King and Kitchener always appears with the fourth level of Perry and the second level of Garrison, etc.), this did not occur. This may be tentative evidence that the four frames capture four unique and different qualities of student thinking that are not simply a correlate of the other frameworks.

By focusing only on the King and Kitchener [7] and Perry [8] frames, an interesting pairing was clarified. Over a third of the postings (33.4%) coded by all four frames captured the request for and response from students of personal experiences and beliefs (KK3 and P7). On the other hand, the Garrison [2] and Bloom [9] frame did not capture this personal nature of postings. In other words, if faculty were in need of a framework that captured students’ personal stories or references to their own lives, King and Kitchener [7] or Perry [8] would be a more preferable choice than Garrison [2] or Bloom [9].

Another interesting finding resulting from looking only at the Garrison [2] and Bloom [9] frames was the emphasis on analysis. Almost a third (or 32.5%) of the postings were related to requesting analyses and responding with analyses that were either exploring or integrating ideas. Again, if faculty were most interested in assessing whether contributions made to an online discussion were largely at the analysis level, Garrison [2] or Bloom [9] would be preferable frameworks.

These patterns reveal insights into the nature of the online discussions and the students. Of the 278 individual postings, 45.3% were at levels five through seven of the King and Kitchener model, 100% were at levels five through nine of the Perry model, 52.2% were at the two highest levels of the Garrison model, and 54.3% were at levels four through six in Bloom’s taxonomy. Not surprisingly, the discussions elicited mid- to high-level responses—as defined by the frames—which one would suspect would be appropriate for doctoral-level students in a doctoral-level class. And while these may be the results for the current set of students, one can see that the frames could also capture younger students’ thinking among the lower and middle levels of the frameworks. In other words, these four frames may be suitable for a range of student abilities and ages.

V. IMPLICATIONS

The answer to the overriding research question, “will these four frameworks be useful and usable?” is a qualified yes. This section will discuss the five lessons taken from this analysis of online discussions as well as from reflection upon the frames and their usefulness for analyzing online discussions. The discussion concludes with two cautions that might affect the application of these and other frames to future analyses of online discussions.

First, it appears that each frame has value. Each frame focuses attention on some particular aspect or quality of the student and his or her thinking as captured in the posting, whether it be reflective judgment or critical thinking. Second, there may not be one best frame, or perhaps one frame might be better suited for a particular discussion or a particularly set of learning objectives. That may indicate a need to develop multiple frames for analyzing online discussions intended to address different learning situations. Faculty may need to use a particular frame in one situation, and another in others, depending on the goals of the discussion or learning situation. Or, in other words, Goethe’s position that “There is nothing insignificant in the world. It all depends on the point of view” may apply just as well to the point of view—or frame of analysis—taken when analyzing online discussions.

Third, Perry [8] was the most difficult framework to use and the distinctions between levels, however relevant, were often difficult to discern in individual postings or even several postings by the same student. This may preclude it being used on a more frequent basis and/or by average faculty. Fourth, the King and Kitchener [7] framework was less difficult to apply, although its use was made easier in some instances when a student wrote a fairly long posting on a topic. It would likely not be appropriate in situations where postings were short or students were not in the habit of explaining their reasoning. Both King and Kitchener [7] and Perry [8] are classification schemas for an individual and not a posting to a discussion, so both frameworks might work better to evaluate online discussions if the discussion were specifically set up to generate a particular reflection or if the student was encouraged to prepare a post that was sufficiently long to give a better sense of the student’s reasoning. In any case, while one student might be consistently at one developmental level in the Perry or King and Kitchener framework, an individual posting to an online discussion might be at a higher (or lower) level, and it would be well to remember that the level of a specific posting (or even several postings) may or may not indicate the student’s predominant developmental stage.

Fifth, as concluded by Meyer [1], the type of triggering question (if we may borrow that term from Garrison [2]) may generate the level of response from other students. Questions created to trigger personal stories did so, and questions targeted to elicit information or higher-level analysis did so; for faculty, the solution to raising the level of online discourse may be more faculty intrusion by setting the discussion’s agenda or actively moderating the discussion, or it may mean training and rewarding students to operate at higher levels. Irrespective of the tendency for subsequent postings to mirror the level of the original posting, many individuals do in fact respond at a higher or lower level. This is an interesting phenomenon and worth exploring further, and may indicate the student’s primary or most comfortable level of response or a response that the group must pass through (i.e., information) in an effort to develop higher-level analyses. Thus, it might be worthwhile to use these frames to analyze the ebb and flow of online discussions as a group effort, rather than focusing on the individual postings as a reflection of the student’s level of thought.

Two important cautions are perhaps in order. While the use of each frame made a contribution to the faculty person’s assessment of student thinking that occurred in the online discussion, there is a danger that a posting might become colored by the point-of-view and values of the frame in an effort to find meaning validated and interpreted by the frame. In other words, the postings and analyses may come to relate more to the frame than to the actual thought expressed. The second caution is to realize that each frame focused the analysis at the same time it eliminated other perspectives. It is like a lens that filters out certain frequencies of light: in time, only blues are seen. This might argue for regular use of a variety of frameworks, in order to keep the analyst and analysis free from mistaking the world for the lens. This might also prevent one frame becoming the only appropriate form of analysis, avoiding Maslow’s caution that “To the man who only has a hammer in the toolkit, every problem looks like a nail.”

Despite these cautions, these frames have added value to the understanding and analysis of online discussions. In fact, additional frameworks are probably needed. One that has been identified in the research literature is a framework for assessing the social presence [11] of individual postings and/or the ability of students to contribute in such a way as to make their personalities come to life in their postings. Another that is needed is a way to assess how a group conversing online works as a group, how it works together to develop an understanding of and solutions to a problem. And if the use of King and Kitchener [7] and Perry [8] has been helpful, there may be other frames developed prior to the explosion of online education that can be usefully applied to understanding the online discussion.

VI. REFERENCES

1. Meyer, K. A. Face-to-Face Versus Threaded Discussions: The Role of Time and Higher-Order Thinking. JALN 7(3): September 2003. Available online: http://www.sloan-c.org/publications/jaln/v7n3/pdf/v7n3_meyer.pdf.
2. Garrison, D. R., Anderson, T., and Archer, W. Critical Thinking, Cognitive Presence, and Computer Conferencing in Distance Education. The American Journal of Distance Education 15(1): 7–23, 2001.
3. Drops, G. Assessing Online Chat Sessions. Online Cl@ssroom, 1–8, April 2003.
4. Pickett, N. Rubrics for Web Lessons, n.d. Available online: http://edweb.sdsu.edu/triton/july/rubrics/Rubrics_for_Web_Lessons.html.
5. Edelstein, S., and Edwards, J. If You Build It, They Will Come: Building Learning Communities Through Threaded Discussions. The Online Journal of Distance Learning Administration 5(1): Spring 2002. Avaliable online: http://www.westga.edu/~distance/ojdla/spring51/edelstein51.htm.
6. Roblyer, M. D., & Ekhaml, L. How Interactive Are YOUR Distance Courses? A Rubric for Assessing Interaction in Distance Learning. The Online Journal of Distance Learning Administration 3(2): Summer 2000. Available online: http://www.westga.edu/~distance/roblyer32.html.
7. King, P. M., and Kitchener, K. S. Developing Reflective Judgment. San Francisco, CA: Jossey-Bass, 1994.
8. Perry, W. G., Jr. Forms of Ethical and Intellectual Development in the College Years: A Scheme. San Francisco, CA: Jossey-Bass, 1999.
9. Bloom, B. S., and Krathwohl, D. R. Taxonomy of Educational Objectives: The Classification of Educational Goals. New York: Longmans, Green, 1956.
10. Krumme, G. Major Categories in the Taxonomy of Educational Objectives (Bloom 1956), n.d. Available online: http://faculty.washington.edu/krumme/guides/bloom.html.
11. Gunawardena, C. N., and Zittle, F. J. Social Presence as a Predictor of Satisfaction Within a Computer-Mediated Conferencing Environment. The American Journal of Distance Education 11(3): 6–26, 1997.

VII. ABOUT THE AUTHOR

Dr. Katrina Meyer is currently assistant professor of educational leadership at the University of North Dakota specializing in online learning and higher education. She is the author of Quality of Distance Education: Focus on On-Line Learning, a 2002 publication of the ASHE-ERIC Higher Education Report Series. For over three years, she was Director of Distance Learning and Technology for the University and Community College System of Nevada. Prior to this, she served over 8 years as Associate Director of Academic Affairs for the Higher Education Coordinating Board in the state of Washington and was responsible for technology planning and policy related to online learning. She can be reached at the University of North Dakota, P.O. Box 7189, Grand Forks, ND 58202; phone: (701) 777-3452; or email: katrina_meyer@und.nodak.edu.