ABSTRACT:Gateway tests provide a means of assuring that students in reformed precalculus and calculus courses acquire the algebraic and computational skills needed in courses following these, while allowing the focus of the course to be on the conceptual understanding intrinsic to the reformed courses. However, logistical difficulties plague the pencil-and-paper administration of these tests, significantly decreasing their usefulness. To address these difficulties we have implemented an on-line version of the gateway tests which allows students to practice the skills on their own and take the test for a grade in a proctored environment. We report on the details of our implementation and assessment of our system, including our administration procedures and lab use. Our assessment included evaluation of students' perceptions of the effectiveness of the program and their actual skill acquisition, and showed that the students' skills improved in the course of the on-line gateway testing.
KEYWORDS: Gateway testing, on-line testing, skills testing, assessment.
INTRODUCTION
In this paper we report on the implementation issues for and results of the assessment of a web-based testing program designed to promote students' acquisition of algebraic and computational skills in precalculus and calculus courses at the University of Michigan. We first describe the context in which this program was implemented, including the historical background of reform at the University. Then we describe the program itself, including information on staffing and required computer resources. We next show the results of our assessment to determine its usability and effectiveness. Finally, we note some extensions to the system that we have put into place. This project was supported by the NSF CCLI Adaptation and Implementation grant program, grant #DUE-0088264, for which the second author (Megginson) was the Principal Investigator, and the implementer of the original paper-based gateway program at the University of Michigan described below. The first author (LaRose) conducted the development and implementation of the on-line gateway system.
CALCULUS REFORM AND GATEWAY TESTING
The University of Michigan has since 1994 taught precalculus and calculus using a reformed textbook, technology, and cooperative group work [I]. In these courses there is a strong emphasis on mathematical concepts rather than on the skill-acquisition that is often prominent in more traditional courses. For example, our midterm arid final exams in calculus include no questions of the standard type "find the [anti] derivative of the function f(x) = . . ." However, even while we believe that our primary focus in these courses should not be on skill acquisition, we recognize that mathematicians, subsequent mathematics courses and courses in other disciplines all expect that students passing these courses will emerge with a well-defined skill set. Therefore, to ensure that students acquire these skills, we include gateway tests in the three reformed courses-precalculus, calculus I and calculus II. We currently have entrance gateway tests covering prerequisite skills for each of precalculus, calculus I and calculus II, and differentiation and integration gateways for calculus I and calculus II respectively. An example of each of these appears in the Appendix.
Gateway tests are skill-based tests on which students must be able to complete a set of 7-12 problems with very few (1-2) errors. Students are allowed to retake the tests throughout a several week testing period until this is accomplished. They are, however, limited to one attempt each day, which encourages them to stop and take the time to figure out what they missed before retaking the test. Because the tests are to prove technical facility with the skills being tested, they have a time cap of 20 minutes. Failure to pass a test by the deadline results in a reduction in the student's final letter grade for the course.1 This reduction is between one-third and a full grade point (per gateway), as determined by the course-coordinator. We have found, however, that almost all students who do not pass the gateway are already doing failing work in the remainder of their coursework (individual and team homework, midterms and final).
The high standard required for a passing grade and allowance for multiple attempts mean that the gateway tests are more of a formative than summative assessment tool. Furthermore, because the gateway tests evaluate a specific skill set to an exacting standard we believe they are more effective than the alternative of including similar questions on the midterms or final, where missing one question would be indistinguishable from other errors on the exam.
As originally implemented, gateway tests were administered as pcnciland-paper tests in the department's Mathematics Tutoring Center. While this was preferable to requiring course instructors to manage them, it proved logistically difficult to sustain. At the University of Michigan we enroll over 1500 students in calculus I in the Fall semester alone, with the result that as the deadline for the gateway test approached the demands on the Tutoring Center frequently exceeded its capacity: the number of tests that needed to be given, graded, processed and returned precluded this happening in a timely manner. As a result there were lines to get into the Tutoring Center and the length of time required for the tutors in the Center to be able to grade and return the tests increased from a day or two to several days or a week.
THE ON-LINE GATEWAY TEST
To address these logistical difficulties we arc now offering all of our gateway tests on-line. This replaces the paper test with an on-line form into which students enter their answers after working out the problem by hand. When they have completed the test they submit it and the testing software automatically and immediately grades it and returns the result to the student. Because the test is on-line it is also possible for students to practice the test before coming to the department to take it for a grade.
When a student registers for one of the courses using the gateway tests, login data are created for him or her on the gateway server. In the week or so preceding the official gateway testing period the instructors in the course distribute this data, which include a login name and password, to the students in their classes. The students are then able to practice the gateway test from their dorm rooms or other University computer labs before being able to take the test for a grade. Once the testing period has begun, students come to a proctored lab in the mathematics department where their identities are verified arid they then take the test in that controlled environment. Only those tests taken in the proctored lab count towards passing the gateway.
The actual gateway tests that are given on-line are drawn from testbanks that are similar to, but larger than, those which were used for the original pencil-and-paper test. Each testbank consists of a number of topics, each containing a large number of questions. A test is generated from the testbank by randomly selecting a predetermined number of questions from each topic. These are presented to the student in any order. Thus each student gets a different test every time he or she logs in to the system.
Each instructor is also given a login name and password for the system, which allows access to data showing which of the students in his or her class(es) have taken the test, and of those, which have passed. Because this interface to the system is also on-line, instructors are able to get instantaneously updated pass/fail data at any time and from any location.
IMPLEMENTATION TIMELINE
