Quality Enhancement Plan
GOAL 4: The College will centralize the online assessment process to include a Director of Research, Assessment, and Planning to oversee the development, collection, analysis, feedback, distribution, and use of results.
As the number of objectives associated with this goal (thirteen) attest, assessment is an integral and essential component of the College's plan to achieve the highest possible quality of online instruction. All of the subcommittees considered assessment in varying degrees and for differing aspects of online instruction, but it was the Assessment of Online Learning subcommittee which had assessment as its sole charge. This goal represents one of their foremost conclusions.
As at least six of the objectives under this particular goal call for the College specifically to compare certain characteristics or aspects of online and traditional courses. These comparisons are in place because the institution assumes that the assessment process will yield results that benefit both methods of course delivery. Objective 4 below, for example, compares attrition and retention rates; Objective 6 compares student demographic data; Objective 7 calls for the development of surveys of online and traditional courses to measure the same concepts; Objective 12 calls for the analysis of student performance in both online and traditional courses. The assessment activities called for by the QEP which compare aspects of online and traditional instruction will provide CVCC with information that will improve learning outcomes in both online and traditional courses. As a result, students, instructors, and the entire institution will be better served and the College better able to fulfill its mission.
The Learning Outcomes subcommittee defined assessment as an ongoing process aimed at understanding and improving student learning. The subcommittee agreed that assessment needs to occur at three separate levels to provide the most comprehensive data: instruction, program, and institutional system. By conducting various kinds of assessments at all three of these levels, the institution will have the kinds of data it needs to gauge progress, track changes, and monitor the directions online learning is taking. This particular goal is concerned with assessment at the program and system level, thus the identification of the Director of Institutional Research, Assessment, and Planning as the lead figure who will oversee the activities directed by this goal. (Assessment at the instructional level will be discussed in greater detail below.)
The need for centralizing the process of collecting information about online and traditional courses became very clear through the contacts with other colleges and the survey of the literature. The colleges that provided the subcommittee with information regarding this topic (including Cochise Community College, Florence-Darlington Technical College, Forsyth Technical Community College, Guilford Community College, and Western Piedmont Community College) have developed a centralized system for collecting the data needed to evaluate both online and traditional courses. The numerous articles dealing with the evaluation of the two methods of course delivery also showed the necessity of having a centralized collection process. The many colleges discussed in these articles simply could not have achieved the level of analysis they reached without having some type of centralized system for collecting the information.
Objective 1: Year One: Centralize all stages of online assessment.
CVCC currently has in place all of the components to assess online courses. However, this process is not centralized. Each of the academic divisions has part of assessment; IT also has a part. These disparate pieces need to be tied together, and this objective makes both concrete and immediate the intent of the College to do so.
Objective 2: Year One through Five: Revise the use of a single online course evaluation instrument by students.
This objective is another result of the findings of the Assessment of Online Learning Subcommittee. The literature and examination of survey instruments developed by institutions or purchased from testing agencies indicate that a variety of survey designs exist. Institutions such as those in the Kentucky Virtual University System through the Kentucky Technical and Community College System (KCTCS) and CVCC use one long comprehensive survey instrument that covers many areas. (CVCC's assessment document comprises some eleven pages and incorporates 65 questions.) KCTCS assessed areas such as Introductory Questions, Choosing Courses, and Demographics. On the other hand, Cochise College, University of Phoenix and Guildford Technical College use separate short surveys to assess specific areas such as Online Course Evaluation and Student Satisfaction. (See Appendix C, figure 2.) After studying these instruments, subcommittee members agreed that CVCC should develop separate evaluation instruments instead of continuing to use the one comprehensive document.
Most institutions incorporate common factors in their survey instruments. These instruments are designed to determine if learning is taking place at the instructional level and if students are being given the support they need to succeed. After examining the literature and observing the instruments, the subcommittee prepared a list of criteria that they determined were important to be included in survey instruments:
- Faculty communication with students including prompt feedback
- Clear course objectives
- Meeting course objectives
- Alignment of course objectives and learning outcomes
- Collaboration among students and faculty
- Different learning styles addressed through a variety of assignments
- Student satisfaction
- Availability of technical support
- Easy navigation of website
In addition to the above criteria, the subcommittee determined from the survey instruments observed that demographic data and space allocated for students� general comments should be included on at least one survey instrument being used by CVCC. Demographics should provide additional information that will facilitate the measurement and comparison of traditional and online learning outcomes. A section for general comments will encourage students to give additional information that can be used to improve courses and programs.
The criteria included in the instruments are mostly supported in an article published by Michigan Virtual University which identifies seven principles of effective teaching while focusing on evaluating online courses. Members observed these principles in the surveys being used by Brevard Community College, Franklin University, Kentucky Virtual University, University of Phoenix, St. Petersburg College, and the North Carolina Community College System. The seven principles are as follows:
Objective 3: Year One through Five: Develop and implement a faculty satisfaction survey for online instruction.
- Good practice encourages student-faculty contact
- Good practice encourages cooperation among students
- Good practice encourages active learning
- Good practice gives prompt feedback
- Good practice emphasizes time on task
- Good practice communicates high expectations
- Good practice respects diverse talents and ways of learning (Graham et al)
This objective reflects further findings of the Assessment of Learning Outcomes subcommittee. CVCC does not currently have any kind of assessment instrument designed to gather this kind of information. Especially influential to the subcommittee was Elements of Quality Online Education by Janet C. Moore. She acknowledges the importance "of a quality framework with practices that show institutional commitment to learning effectiveness, cost effectiveness, access, faculty satisfaction, and student satisfaction" (3). Adding this particular instrument to the rest of the assessment package will allow the institution to systematically gather data which before was available only anecdotally. These data will provide a valuable new perspective into online teaching from the instructor side of the desk.
Objective 4: Year One through Five: Develop and implement an analysis of attrition/retention rates by comparing online with traditional courses.
According to the data the College has been able to collect so far, the overall attrition rate for online courses is higher than that for face-to-face classes. This rate varies, however, from class to class for reasons that so far remain unanalyzed. The institution is aware that rates of attrition and retention are significant measures of student satisfaction that are integral to creating successful learning outcomes. The research further suggests that indicators used to examine attrition/retention rates may include family size, family income and educational background, age, gender, education, placement scores and ethnicity (Moore 44). PeopleSoft, the new student information software system, will allow CVCC to create a database of many of these demographic elements so that that they can be used to generate data that determine rates such as retention and attrition. Data collected from these analyses and other assessment instruments can be analyzed and used to (a) change or modify programs and/or courses, (b) compare learning outcomes among courses and programs, and (c) compare online and traditional completion rates and learning outcomes. Ultimately, a collection and study of these statistics will lead to changes and improvement of learning outcomes at CVCC.
Objective 5: Year Two: Evaluate by means of a survey Objective 1 under Goal 7 regarding delivery of Student Support Services.
This objective will be alluded to again in the discussion of Goal 7 below. In summary, however, Goal 7 commits the College to provide similar student support services to individuals taking online classes as are provided students in traditional classes. These student support services are to be phased in over the five-year course of the QEP. Objective 1 comprises Academic Records, Admissions, Registration, Exam and Test Proctoring, Tuition and Fee Payment, Tutoring, Technical Support, and Bookstore. This objective calls for the creation of a method to ascertain to what extent this portion of Goal 7 is being achieved.
Objective 6: Year Two through Five: Determine the demographic characteristics of online learners compared with traditional learners in baseline databases.
As has been explained under Objective 4 immediately above, collecting student demographic data can be valuable in helping to determine reasons behind attrition and retention rates. In order to effectively assess differences between online and traditional methods of teaching, the College needs to also analyze the differences among our students. Currently, for example, adult learners comprise 80% of web students (Newcombe). Some of the studies that claim online courses to be as effective (or more effective) as traditional courses fail to take into account differences in the student populations (Hiltz, Zhang, and Turoff 14-15, 18-22). However, researchers maintain that differences between students in online and traditional courses can significantly affect the levels of success. They see demographic data as strongly affecting student performance. (Grayson, MacDonald, and Saindon; Parker and Germino). Another key area highlighted in the research is that students have a variety of different learning styles that can affect their performance. Some of the studies examined reveal that since online courses frequently employ instructional methods that differ from those used in traditional courses, determining the learning styles of the different students could be a useful indicator as to which type of learning environment would work best for the individual student. (Grayson, MacDonald, and Saindon; Johnson et al). According to some research, for example, although the classroom behaviorist model may be used for online learning, the constructivist model (the instructor acts as facilitator) is better suited to the online environment (Hazari; Driscoll; Seels; Ryan et al; Polichar & Bagwell, 2000).
Demographic data can yield much, but as the timeline for this objective makes clear, the process of collecting and analyzing the data will be an ongoing process. The institution will need to determine which specific demographic data items are most valuable and in what contexts. The multi-year timeline for this objective will provide the necessary time for gathering data, establishing baselines for the baseline database, and then analyzing what information the data reveal.
Objective 7: Year Two through Five: Develop and implement online and traditional course surveys measuring the same concepts.
Much research has been conducted on what works pedagogically in the traditional classroom and work is currently being conducted to gather this same information for online courses. This objective represents CVCC's effort to do the same for our particular courses and students. Research examined by the Assessment of Online Learning subcommittee found that Brevard Community College in Florida, for example, has been compiling data that compares the grades and retention rates of students in the baseline online and traditional classes to data compiled from students taking online courses to examine attrition and retention rates. The surveys that will be developed at CVCC will allow the College to gather similar information plus additional data. These surveys will allow CVCC to help determine what kinds of students take and succeed in an online course and compare their characteristics to those of students in traditional classes.
Objective 8: Year Two through Five: Administer surveys such as student satisfaction for courses where online and traditional students are taught by the same instructor.
Measuring student satisfaction in this fashion will provide another perspective into what works--and how--in traditional and online courses. Examining data from classes taught by the same instructor will permit the survey to limit the number of variables assessed and focus on those elements which delineate online from traditional classes. At the same time, these surveys should ideally identify how the two delivery methods are similar. These data may also suggest that strategies or methodologies employed in one delivery mode can be beneficial in the other. For example, anecdotal evidence gathered at CVCC seems to suggest that students appreciate and are growing to expect that traditional classroom classes have an online component. Students most often mention the helpfulness of lecture notes and study guides that some teachers in the traditional classroom make available on a web site.
Objective 9: Year Two through Five: Evaluate the orientation methods for online courses through the use of online student satisfaction survey.
When CVCC first began offering online classes back in fall 1998, most instructors found that offering an on-campus orientation was beneficial both to them and their students. Such a meeting allowed instructors to associate a face with the names on their class roster; more important, this meeting allowed the instructors to go over the mechanics of their online class. Instructors could show students how to access the course on the Internet, where to find course assignments, and how to submit them. In those days too, different online formats proliferated. CVCC had courses in FrontPage, WebCT, and Blackboard.
Many CVCC students were not sophisticated computer users, and these orientation sessions gave them the knowledge they needed and permitted the instructor to disseminate essential course material efficiently. However, with each passing term students gained computer knowledge, and some instructors found orientations less necessary. Now that the VCCS has purchased and installed system-wide Blackboard servers, all online courses share a common platform, and as students become familiar with Blackboard, this may further lessen the need for orientation sessions for online courses. This objective will allow the institution to gather data to determine the best orientation practices and implement these practices throughout the online curriculum.
Objective 10: Year Two through Five: Review and evaluate Objectives 2-12 annually and implement improvements if indicated and disseminate findings.
This objective seems self-explanatory. It commits the College to continue to examine and refine the various assessment instruments and policies associated with the Quality Enhancement Plan to insure their maximum effectiveness.
Objective 11: Year Three: Evaluate by means of a survey Objective 2 under Goal regarding delivery of Student Support Services.
This objective is analogous to Objective 5 immediately above. As discussed under that objective, Goal 7 (see below) commits the College to offer online the full complement of student services currently available on campus. Various student services are to be phased in over the life of the QEP, and this particular Objective is designed to assess the online efficacy of the following student service functions: Academic Advisement, Financial Aid, Learning Resources, Disability Services, and Placement Testing.
Objective 12: Year Three through Five: Analyze and compare student performance in online and traditional courses.
This objective represents yet another facet of the assessment process that will provide much of the information that guides this QEP. By looking at student performance data - grades certainly, but also other aspects that might include competencies or skills, test scores, written work, et cetera - the institution will be in a position to determine how the two course delivery methods compare in terms of learning outcomes. The question has been raised, especially from those strong adherents of online instruction, whether online education might prove to be a more effective way of teaching than the traditional classroom. The data gathered as a result of this objective will speak to that particular question, at least as far as CVCC students are concerned, and should offer interesting insights into the nature of how (and what) students learn. The best outcome for such investigations and analyses of the resulting data might be to provide instructors with the strategies that produce the most effective learning; as a result, students who opt for either mode of instructional delivery can achieve the maximum benefit each has to offer.
The results from the analyses of the data gathered to meet this objective can be combined with additional research already available to help instructors design new courses and modify existing courses to make them more effective. Accordingly, their characteristics should be considered when creating, editing, or reviewing web courses (Darkenwald & Merriam; Weiss, Knowlton, & Speck).
Objective 13: Year Three through Five: Survey and analyze reasons for withdrawal from online classes as compared to traditional courses.
This objective is a corollary to Objective 5 immediately above. The previous objective will look at attrition and retention rates; this objective, however, will gather data that specifically address the reasons students withdraw. While some of the reasons will doubtless be common to both online and traditional classes, it will be interesting to see if certain reasons are more common to one or the other. If this turns out to be the case, the institution will be able to offer more effective guidance to students who are planning their curricula, and instructors will be better able to assess their students' progress and counsel them if they fall into any of the at-risk categories.