Assessing & Comparing Outcomes in a Lab-Based Science Course Delivered Traditionally and Online
Tod Treat, Parkland College
II Background
IV. Results
V. Implications for the Future.
Online instruction at community colleges has grown tremendously over the past several years. At Parkland, for example, 5 courses in 1995 has grown to over 130 sections this year. This growth has been accompanied by increased questions about the effectiveness of online courses. Online instruction has a great deal of promise, but is also subject to criticism. Some feel that it does not live up to its promise of solving teaching learning issues, such as increased flexibility Others are concerned that the technology itself creates a barrier that requires additional understanding for participants (both faculty and students). “Online instruction also threatens to commercialize education, isolate students and faculty, and may reduce standards or even devalue university degrees (Johnson, Aragon, Shaik, and Palma-Rivas, 2000).” More needs to be known about how student experience differs from face-to-face environments and how outcomes are developed and measured.
Problems to be explored:
Parkland College's Academic Assessment plan is in the process of being implemented.
Books that are excellent in this area include Classroom Assessment Techniques, Angelo and Cross (1993) and Effective Grading, Walvoord and Anderson (1998).
Research has been done comparing online to face-to-face instruction. Schutte (1997) divided his class in two, one face-to-face the other online, and showed that instruction was “improved” in online environments. LaRose, Gregg, and Eastin (1998) compared learning outcomes and found no significant difference. In the latter study, however, one section had face-to-face lectures while the other section listened to prerecorded audio and accessed outlines, etc online. In both cases, the methodology was flawed by differences in the structure of the sections these students accessed.
More studies have looked at satisfaction.
While satisfaction is high according to these studies, the reasons vary.
Enockson (1997) found that students were satisfied because of the
flexibility and responsiveness available in online courses.
Johanson (1996) found that students satisfactions increases when (1) the
technology works and flows well, (2) the course is learner-centered, (3) the
instructor acts as “facilitator and coach”, and (4) there is built-in
flexibility. Debourgh (1998) found that satisfaction depends on the instructor
more than the technology.
As degrees are offered via nontraditional methods,
accountability has become much more important. Leadership in educational technology will focus on seven
areas, which I have adapted from an article by Merisotis and Phipps that focuses
on research comparing distance and traditional classrooms (Merisotis and Phipps,
1999)
1. Research thus far has focused on courses not programs. Some sort of standardized assessment of outcomes at the programmatic level will need to be developed and used to strengthen programs and courses in an increasingly competitive market. Currently, students “shop” for online courses. A strong programmatic approach to distance learning will bring students to online education and keep them here. Penn State has modeled this approach with their innovative turf science program.
2. Research hasn’t looked at how different groups of students are learning in alternative delivery courses. Much has been made of the changing demographic in education, but we still don’t know who we will be serving, what access they have to the technology, and what their specific needs are.
3. Research hasn’t explained (or solved) the higher course dropout rates of students in alternative delivery courses. Ultimately, the health of any distance education program will be dependent on it’s ability not only to recruit students, but to retain them.
4. Learner characteristics and learning styles have not been correlated to the use of particular technologies. Research on this aspect of online course development has shown improvement of student attitudes and satisfaction when online courses are sensitive to learning styles. I would encourages and enable faculty to consider instructional design and technology choices that take such characteristics into account.
5. Research has not focused on the synergistic effects of certain technologies and formats. I suspect that most successful courses have chosen technologies that integrate in a way that makes them more effective.
6. The research lacks a theoretical framework. While Merisotis and Phipps are focusing on the replicability of research, I would argue that a theoretical framework is necessary as a foundation for growth and success in alternative delivery.
7. The research hasn’t addressed digital libraries (and other student support services). It is clear that successful online delivery cannot be limited to courses. Student support services, such as online tutors, library services, registration, help-lines, and other services are equally as important as the instruction that takes place in the course.
Course Format:Lectures:
Live, on-tape, PP slides, streaming videoLabs:Outcomes determine flexibility. A mixture of on-campus and at-home experiments.Assessments/discussions:All sections are combined and interact in WebCT. Weekly discussion questions encourage interaction.Summative Assessment:Portfolio-based to encourage revisions and questions
Eight students were enrolled in the on campus course and eight were enrolled in the online course. Pretesting of basic science concepts was used to compare the incoming level of knowledge of students. Students in both courses accessed course information via WebCT, performed discussions together online, took the same quizzes/tests, and had the same portfolio due. Labs were correlated according to outcomes and were the same wherever it was sensible to do so (Appendix A).
Statistical analysis was not performed due to lower power caused by small sample sizes.
Pre- testing
Basic concepts were tested by a twenty question survey covering topics on scientific method, basic chemistry, biology, physics, etc. The results indicate that there may be some self-selection resulting in a better background among the online students.
|
|
Percent correct |
|
On campus |
68.3% |
|
Online |
76.0% |
Content
The online students performed better on both the final exam and the portfolio scores. Some of this, however, is attributed to the better background that these students brought to the course. From the results, the appears to be no significant difference between online and on campus learning in Chemistry 104.
|
|
Portfolio Average |
Final Exam Average |
|
On Campus |
261/400 |
77.7 |
|
Online |
301/400 |
79.0 |
Attitude (see Appendix B)
Questions were asked about general education objectives and general attitudes about science using Likert scales. The difference in attitudes towards general education were significant. On campus students felt that they had improved on the general education objectives. Attitudes toward science were not significantly different.
V. Implications for the Future.
Assessment works as a tool for comparing online and on campus courses. Online students are getting a comparable education, but may have lower levels of satisfaction. More assessment needs to be done, but this pilot study indicates that it can be done in a way that is meaningful to instructors. Course management software, such as WebCT, can be very helpful in this regard.
Johnson, S.D., Aragon, S.R., Shaik, N., and Palma-Rivas,
N. (2000). Comparative analysis of learner satisfaction and learning outcomes
in online and face-to-face learning environments.
Journal of Interactive Learning Research.
Merisotis, Jamie, and Phipps, Ronald. “What’s the
Difference? Outcomes of Distance vs. Traditional Classroom-Based Learning.” Change,
May/June 1999, pp 13-17.
Schutte, J.G. (1997). Virtual teaching in higher
education: The new intellectual superhighway or just another traffic jam.
Available: http://www.csun.edu/sociology/virexp.htm
Appendix A: Comparison of Laboratory Outcomes
|
Labs for Chemistry 104 |
|||||
|
Topic |
Labs |
Course specific Objectives |
Natural Sciences Laboratory Objectives | Lab Instruction Style |
Degree of Similarity Online vs. Oncampus |
| Origins | Lab Safety Activity |
|
? | Informational/Activity | Same |
| Connections | Density Layers |
|
A.1,2,7
B.1,3,4 C.1,2 |
Expository/Wet | Objective 3 is not obtainable online. |
| Bonding | Thin Layer
Chromatography (Internet Activity- Paper Chromatography) |
|
A.1,2,5,7
B.1,2,3 C.1,2,3 |
Discovery/Wet | Same |
| Recycling | Municipal
Waste |
|
A.3,4,5
B.1,3 C.2 |
Expository/Activity | Same |
| Recycling | Energies of Neutralization |
|
A.1,2,4,5,7
B.1,2 C.1,2 |
Expository/Wet | Same |
| Alternative Energy Sources | Solar Lab |
|
A.1,2,3,4
B.1 C.1,2,3 |
Discovery/Wet | Same |
| Alternative Energy Sources | Modelling Project |
|
A.2
C.1,2,3 |
Expository/Simulation | Same |
| Molecular Structure | Polymers (Internet Activity- Plastics Lab) |
|
A.2,3,4
B.1,2 C.2 |
Expository/Wet | Same |
| Nutrition | Nutrition Lab |
|
A.3,7
C.2,3 |
Informational/Simulation | Same |
| Nutrition | Food Lab |
|
A.1,2,3
B.2 C.1,2,3 |
Discovery/Wet | Same |
| Chemistry of Life | Virtual
Food Lab (do prelab,exp't, and postlab) |
|
A.4
B.1,3 C.1,2 |
Expository/Simulation | Same |
| Drugs and Toxicology | (Internet Activity-Drug Hunt Discussion) |
|
C.2,3 | Informational/Activity | Same |
| Forensic Chemistry |
(Internet Activity-Murder Mystery) |
|
A.3,6
B.1,3 C.1,2,3 |
Online-Discovery/Activity
Oncampus-Discovery/Web |
Similar outcomes, but quite different |
| Forensic Chemistry | DNA fingerprinting |
|
A.1,2
C.1,2,3 |
Informational/Simulation | Same |
| Category |
Question |
Oncampus |
Online |
| Gen Ed |
To what extent did this course help you to, "demonstrate [your] ability to read, write, listen, and speak effectively" [1-improved a lotà4-it hurt me] |
2.5/4.0 |
1.6/4.0 |
| Gen Ed |
To what extent did this course help you to, "demonstrate [your] ability to think critically, which includes collecting facts and making decisions based on them, and solving problems, using methods of critical and scientific inquiry" [1-improved a lotà4-it hurt me] |
1.9/4.0 |
1.6/4.0 |
| Gen Ed |
To what extent did this course help you to, "demonstrate [your] ability to compute and to think and express themselves effectively in quantitative terms" [1-improved a lotà4-it hurt me] |
2.0/4.0 |
1.6/4.0 |
| Gen Ed |
To what extent did this course help you to, "demonstrate [your] ability to use technology, especially computer technology, to access, retrieve, process and communicate information" [1-improved a lotà4-it hurt me] |
1.9/4.0 |
1.2/4.0 |
| Gen Ed |
To what extent did this course help you to, "demonstrate [your] understanding of world-wide political, social, and economic issues, historical and geographical perspectives, the internal and external world, and philosophical ideas" [1-improved a lotà4-it hurt me] |
1.6/4.0 |
1.4/4.0 |
| Gen Ed |
To what extent did this course help you to, "demonstrate [your] understanding of the necessity of core values in helping them make ethical, personal, social, and professional decisions" [1-improved a lotà4-it hurt me] |
2.4/4.0 |
2.2/4.0 |
| Gen Ed |
To what extent did this course help you to, "demonstrate [your] understanding of the benefits of diversity in cultures, ideas, perspectives, ethnicity, religion, gender, and sexual orientation for a democratic society" [1-improved a lotà4-it hurt me] |
2.6/4.0 |
2.0/4.0 |
| Attitudinal |
I do at least one of the following: go to science museums, subscribe to a science magazine, support a science or environmental organization. [5-agree stronglyà1-disagree strongly] |
3.0/5.0 |
2.3/5.0 |
| Attitudinal |
I can do at least two of the following accurately: use a microscope, measure volumes, use a scientific calculator, use a pH meter, identify rocks and minerals [5-agree stronglyà1-disagree strongly] |
1.5/5.0 |
1.0/5.0 |
| Attitudinal |
I am very comfortable reading articles or watching shows about science (astronomy, geology, biology, chemistry, physics, etc) [5-agree stronglyà1-disagree strongly] |
3.0/5.0 |
1.3/5.0 |
| Attitudinal |
I understand articles or shows about science (astronomy, geology, biology, chemistry, physics, etc. [5-agree stronglyà1-disagree strongly] |
3.0/5.0 |
2.5/5.0 |
| Attitudinal |
Articles and shows about science confuse me. I do not understand them very well. [5-agree stronglyà1-disagree strongly] |
3.0/5.0 |
3.0/5.0 |
| Attitudinal |
I enjoy science and articles/shows about science. [5-agree stronglyà1-disagree strongly] |
2.0/5.0 |
2.0/5.0 |
| Attitudinal |
I do not care to know how stuff like the body, chemical reactions, or gravity work [5-agree stronglyà1-disagree strongly] |
3.7/5.0 |
3.5/5.0 |