Assessing & Comparing Outcomes in a Lab-Based Science Course Delivered Traditionally and Online


Presented at “Best Practices: Past, Present, and Future”
ICCFA Conference 2000

Tod Treat, Parkland College

I Introduction

II Background

   A. Assessment vs. Research

   B. Issues in Online Education

   C. Chemistry 104

III. Methods of Assessment

IV. Results

V. Implications for the Future.

 


I Introduction

Online instruction at community colleges has grown tremendously over the past several years. At Parkland, for example, 5 courses in 1995 has grown to over 130 sections this year. This growth has been accompanied by increased questions about the effectiveness of online courses. Online instruction has a great deal of promise, but is also subject to criticism. Some feel that it does not live up to its promise of solving teaching learning issues, such as increased flexibility Others are concerned that the technology itself creates a barrier that requires additional understanding for participants (both faculty and students). “Online instruction also threatens to commercialize education, isolate students and faculty, and may reduce standards or even devalue university degrees (Johnson, Aragon, Shaik, and Palma-Rivas, 2000).” More needs to be known about how student experience differs from face-to-face environments and how outcomes are developed and measured.

Problems to be explored:

  1. Can an online section and an on campus section be offered synergistically and with similar structure?
  2. Are the outcomes between online and on campus sections comparable?
  3. Are the attitudes between online and on campus sections comparable?
  4. Can assessment be used to improve teaching and learning in online environments?

II Background

   A. Assessment vs. Research

Parkland College's Academic Assessment plan is in the process of being implemented.  

Working Assumptions

Books that are excellent in this area include Classroom Assessment Techniques, Angelo and Cross (1993) and Effective Grading, Walvoord and Anderson (1998).

   B. Issues in Online Education

Research has been done comparing online to face-to-face instruction. Schutte (1997) divided his class in two, one face-to-face the other online, and showed that instruction was “improved” in online environments.  LaRose, Gregg, and Eastin (1998) compared learning outcomes and found no significant difference. In the latter study, however, one section had face-to-face lectures while the other section listened to prerecorded audio and accessed outlines, etc online. In both cases, the methodology was flawed by differences in the structure of the sections these students accessed.

More studies have looked at satisfaction.  While satisfaction is high according to these studies, the reasons vary.  Enockson (1997) found that students were satisfied because of the flexibility and responsiveness available in online courses.  Johanson (1996) found that students satisfactions increases when (1) the technology works and flows well, (2) the course is learner-centered, (3) the instructor acts as “facilitator and coach”, and (4) there is built-in flexibility. Debourgh (1998) found that satisfaction depends on the instructor more than the technology.  

Research has also been done to determine ways to improve instructional design.  Harrison, Seeman, Behm, Saba, Molise  and Williams (1991) determined that effective design includes four components: instruction, management, telecommuting, and support. Jegede, Fraser, and Curtin (1995) identified more components: interactivity, institutional support, task orientation, teacher support, negotiation, flexibility, technological support, and ergonomics. (Access the references listed above from the Johnson paper.)

As degrees are offered via nontraditional methods, accountability has become much more important.  Leadership in educational technology will focus on seven areas, which I have adapted from an article by Merisotis and Phipps that focuses on research comparing distance and traditional classrooms (Merisotis and Phipps, 1999)

1.        Research thus far has focused on courses not programs.  Some sort of standardized assessment of outcomes at the programmatic level will need to be developed and used to strengthen programs and courses in an increasingly competitive market.  Currently, students “shop” for online courses.  A strong programmatic approach to distance learning will bring students to online education and keep them here.  Penn State has modeled this approach with their innovative turf science program.

2.        Research hasn’t looked at how different groups of students are learning in alternative delivery courses.  Much has been made of the changing demographic in education, but we still don’t know who we will be serving, what access they have to the technology, and what their specific needs are.  

3.        Research hasn’t explained (or solved) the higher course dropout rates of students in alternative delivery courses.  Ultimately, the health of any distance education program will be dependent on it’s ability not only to recruit students, but to retain them.  

4.        Learner characteristics and learning styles have not been correlated to the use of particular technologies.  Research on this aspect of online course development has shown improvement of student attitudes and satisfaction when online courses are sensitive to learning styles.  I would encourages and enable faculty to consider instructional design and technology choices that take such characteristics into account. 

5.        Research has not focused on the synergistic effects of certain technologies and formats.  I suspect that most successful courses have chosen technologies that integrate in a way that makes them more effective.

6.        The research lacks a theoretical framework.  While Merisotis and Phipps are focusing on the replicability of research, I would argue that a theoretical framework is necessary as a foundation for growth and success in alternative delivery.  

7.        The research hasn’t addressed digital libraries (and other student support services). It is clear that successful online delivery cannot be limited to courses. Student support services, such as online tutors, library services, registration, help-lines, and other services are equally as important as the instruction that takes place in the course.  

 

   C. Chemistry 104

Chemistry 104, Chemistry of Everyday Life, ( http://online.parkland.edu/ttreat/che104) is offered in three formats.  Students “cross- fertilize” in the course, leading to a combination of flexibility and increased  increased interaction.  Laboratory experiments are assigned according to desired outcomes
Course Format:

Lectures:

Live, on-tape, PP slides, streaming video
Labs:
Outcomes determine flexibility. A mixture of on-campus and at-home experiments.
Assessments/discussions:
All sections are combined  and interact in WebCT.  Weekly discussion questions encourage interaction.
Summative Assessment:
Portfolio-based to encourage revisions and questions

 

III. Methods of Assessment

Eight students were enrolled in the on campus course and eight were enrolled in the online course.  Pretesting of basic science concepts was used to compare the incoming level of knowledge of students.  Students in both courses accessed course information via WebCT, performed discussions together online, took the same quizzes/tests, and had the same portfolio due.  Labs were correlated according to outcomes and were the same wherever it was sensible to do so (Appendix A).

IV. Results  

Statistical analysis was not performed due to lower power caused by small sample sizes.

Pre- testing

Basic concepts were tested by a twenty question survey covering topics on scientific method, basic chemistry, biology, physics, etc. The results indicate that there may be some self-selection resulting in a better background among the online students.

 

Percent correct

On campus

68.3%

Online

76.0%

Content

The online students performed better on both the final exam and the portfolio scores.  Some of this, however, is attributed to the better background that these students brought to the course. From the results, the appears to be no significant difference between online and on campus learning in Chemistry 104.

 

Portfolio Average

Final Exam Average

On Campus

261/400

77.7

Online

301/400

79.0

Attitude (see Appendix B)

Questions were asked about general education objectives and general attitudes about science using Likert scales. The difference in attitudes towards general education were significant. On campus students felt that they had improved on the general education objectives. Attitudes toward science were not significantly different.

V. Implications for the Future.

Assessment works as a tool for comparing online and on campus courses.  Online students are getting a comparable education, but may have lower levels of satisfaction.  More assessment needs to be done, but this pilot study indicates that it can be done in a way that is meaningful to instructors. Course management software, such as WebCT, can be very helpful in this regard.


Bibliography

 Johnson, S.D., Aragon, S.R., Shaik, N., and Palma-Rivas, N. (2000). Comparative analysis of learner satisfaction and learning outcomes  in online and face-to-face learning environments.  Journal of Interactive Learning Research.

Merisotis, Jamie, and Phipps, Ronald. “What’s the Difference? Outcomes of Distance vs. Traditional Classroom-Based Learning.” Change, May/June 1999, pp 13-17.

Schutte, J.G. (1997). Virtual teaching in higher education: The new intellectual superhighway or just another traffic jam. Available: http://www.csun.edu/sociology/virexp.htm

 

 

 

Appendix A: Comparison of Laboratory Outcomes

Labs for Chemistry 104

Topic

Labs

Course specific Objectives

Natural Sciences Laboratory Objectives Lab Instruction Style
Degree of Similarity Online vs. Oncampus
Origins Lab Safety Activity
  • provide you with a chance to learn more about chemical safety information and to understand standard procedures for students in Parkland’s labs
? Informational/Activity Same
Connections Density Layers

(Internet Activity-Density)

  • To create density gradients by layering common materials of different densities. 
  • To observe the positions taken by solid objects placed into a density gradient and then estimate the density from its position. 
  • To demonstrate the ability to measure the density of common substances within a 10% error. 
A.1,2,7

B.1,3,4

C.1,2

Expository/Wet Objective 3 is not obtainable online.
Bonding Thin Layer Chromatography
(Internet Activity- Paper Chromatography)
  • To learn how chromatography can be used to separate and purify components of mixtures.

  • To understands the basic concepts of mobile phase and stationary phase in chromatography.

  • To learn to make qualitative and quantitative observations and report them cleanly in tables.

  • To practice experimental design 

A.1,2,5,7

B.1,2,3

C.1,2,3

Discovery/Wet Same

 

Recycling Municipal Waste
  • To monitor and analyze your production of selected waste materials, and to compare your production with that of the typical American.
  • To use quantitative methods to do comparisons
A.3,4,5

B.1,3

C.2

Expository/Activity Same
Recycling Energies of Neutralization
  • Learn to make careful measurements.
  • Learn to record data in a systematic and organized fashion.
  • Learn to think about complex chemical systems and why they behave the way they do.
  • Make energy calculations on real experimental data.
  • Relate energy transfer to chemical reactions.
A.1,2,4,5,7

B.1,2

C.1,2

Expository/Wet Same
Alternative Energy Sources Solar Lab
  • Learn about terms and features used in climate research by viewing primary data.
  • Discover differences in energy absorbing features of solar systems.
  • Apply discoveries to design a passive solar house.
A.1,2,3,4

B.1

C.1,2,3

Discovery/Wet Same
Alternative Energy Sources Modelling Project
  • Practice model building as a way to predict and analyze the 3-D structure of molecules. 
  • Learn to appreciate the simplicity and practicality of Lewis dot structures in determining shape. 
  • Learn to explain the behaviors of certain molecules as a function of their shape.
  • Develop an understanding of polymers and how they work.
A.2

C.1,2,3

Expository/Simulation Same
Molecular Structure Polymers
(Internet Activity- Plastics Lab)
  • To connect polymer structures to the universal symbols found on containers.
  • To appreciate the differences in physical properties and chemical structure of polymers.
  • To consider societal issues related to plastic production and disposal.
A.2,3,4

B.1,2

C.2

Expository/Wet Same

 

Nutrition Nutrition Lab
  • To determine how much energy you require on a daily basis by tracking your activities, then comparing that to your nutritional intake based
    on seven days of dietary data. 
A.3,7

C.2,3

Informational/Simulation Same
Nutrition Food Lab
  • To discover, through basic experimentation, examples of scientific principles in the chemical behavior of food.
A.1,2,3

B.2

C.1,2,3

Discovery/Wet Same
Chemistry of Life Virtual Food Lab
(do prelab,exp't, and postlab)
  • To learn basic tests applied to food.
  • To learn to analyze data to determine food composition.
  • To develop skills in analyzing nutritional labels.
A.4

B.1,3

C.1,2

Expository/Simulation Same
Drugs and Toxicology (Internet Activity-Drug Hunt Discussion)
  • To identify common drugs found at home.
  • To learn about the mode of action, structure, and contraindications of drugs.
  • To analyze information presented by the manufacturer.
C.2,3 Informational/Activity Same
Forensic Chemistry

(Internet Activity-Murder Mystery)

  • To apply scientific method to solving of a crime.
A.3,6

B.1,3

C.1,2,3

Online-Discovery/Activity

Oncampus-Discovery/Web

Similar outcomes, but quite different
Forensic Chemistry DNA fingerprinting
  • To learn about how DNA is used and attempt to use this information to solve crimes.
A.1,2

C.1,2,3

Informational/Simulation Same

Appendix B: Survey Questions

Category

Question

Oncampus

Online

Gen Ed

To what extent did this course help you to, "demonstrate [your] ability to read, write, listen, and speak effectively"

[1-improved a lotà4-it hurt me]

2.5/4.0

1.6/4.0 

Gen Ed

To what extent did this course help you to, "demonstrate [your] ability to think critically, which includes collecting facts and making decisions based on them, and solving problems, using methods of critical and scientific inquiry"

[1-improved a lotà4-it hurt me]

1.9/4.0

1.6/4.0

Gen Ed

To what extent did this course help you to, "demonstrate [your] ability to compute and to think and express themselves effectively in quantitative terms"

[1-improved a lotà4-it hurt me]

2.0/4.0

1.6/4.0

Gen Ed

To what extent did this course help you to, "demonstrate [your] ability to use technology, especially computer technology, to access, retrieve, process and communicate information"

[1-improved a lotà4-it hurt me]

1.9/4.0

1.2/4.0

Gen Ed

To what extent did this course help you to, "demonstrate [your] understanding of world-wide political, social, and economic issues, historical and geographical perspectives, the internal and external world, and philosophical ideas"

[1-improved a lotà4-it hurt me]

1.6/4.0

1.4/4.0

Gen Ed

To what extent did this course help you to, "demonstrate [your] understanding of the necessity of core values in helping them make ethical, personal, social, and professional decisions"

[1-improved a lotà4-it hurt me]

2.4/4.0

2.2/4.0

Gen Ed

To what extent did this course help you to, "demonstrate [your] understanding of the benefits of diversity in cultures, ideas, perspectives, ethnicity, religion, gender, and sexual orientation for a democratic society"

[1-improved a lotà4-it hurt me]

2.6/4.0

2.0/4.0

Attitudinal

I do at least one of the following: go to science museums, subscribe to a science magazine, support a science or environmental organization.

[5-agree stronglyà1-disagree strongly]

3.0/5.0

2.3/5.0

Attitudinal

I can do at least two of the following accurately: use a microscope, measure volumes, use a scientific calculator, use a pH meter, identify rocks and minerals

[5-agree stronglyà1-disagree strongly]

1.5/5.0

1.0/5.0

Attitudinal

I am very comfortable reading articles or watching shows about science (astronomy, geology, biology, chemistry, physics, etc)

[5-agree stronglyà1-disagree strongly]

3.0/5.0

1.3/5.0

Attitudinal

I understand articles or shows about science (astronomy, geology, biology, chemistry, physics, etc.

[5-agree stronglyà1-disagree strongly]

3.0/5.0

2.5/5.0

Attitudinal

Articles and shows about science confuse me. I do not understand them very well.

[5-agree stronglyà1-disagree strongly]

3.0/5.0

3.0/5.0

Attitudinal

I enjoy science and articles/shows about science.

[5-agree stronglyà1-disagree strongly]

2.0/5.0

2.0/5.0

Attitudinal

I do not care to know how stuff like the body, chemical reactions, or gravity work

[5-agree stronglyà1-disagree strongly]

3.7/5.0

3.5/5.0