December 2, 1999

Chapter 6 "Improving Assessment with Technology"

Introduction

The focus of the previous sections has been to use technology as one of the teaching and learning strategies employed in a course to meet student learning objectives. This is the most important reason for choosing technology, but some technologies can also be used to suppport and improve assessment efforts. For example, some classroom assessment techniques, discussed in the previous section, were designed to be used at the end of a particular class to help faculty see the progress that students are making on a set of learning objectives. This might improve the quality of the assessment by allowing students more time to reflect on the question or it could help by just making students think about the course and take some action between class periods. Ther are advantages and disadvantages for using technology for assessment that instructors will need to evaluate.

There are several examples of how technology can help with the management and flow of the course. In this section, however, we will focus on the uses of technology that directly relate to the collection of information from students for guiding student learning and making course corrections. The technology can be used to help provide feedback to students.

We feel that using technology to aid in assessment is a sufficient reason on its own to incorporate technology into a course. Independent of any other learning outcome goals and teaching/learning strategy choices, technology is worthwhile just from the point of view of the gains in the quality of assessment.

One way to demonstrate the range of technology options is to give a set of examples from the simplest to more sophisticated applications. Each example illustrates the use of technology and ties to the assessment approach. A more detailed description of the computer programs with template programs are given in the appendix to this chapter. We have described the technologies here in the generic terms (such as threaded discussion group) but many of these are found as components of web course development packages (such as WebCT and many others).

Examples of technology

The example uses of technology can be divided into three basic categories, asynchronous, grading and tracking.

TABLE 1: Three categories of assessment and example technology approaches.

1. Email between faculty and students can be used as a method for obtaining individual written answers to questions and individual responses to student questions. The questions for students can be "broadcast" with a list or a listserv. Student answers can be collected and sorted into separate files using some of the more sophisticated email programs. Email is simple to use and ubiquitous. The major problem with this is that there is no way to handle this efficiently. Most email programs have limited features and are missing some of the word processing and sorting functions. In addition, each email response is handled as an individual document which impedes its use for data analysis and comparisons.

2. Browser forms to email. The next step up in sophistication is to prompt responses from students by having them respond to a form on a web page. This allows a more focused responses by students. This feature can be easily implemented by using a simple cgi that redirects the contents of the form to email (forms to email). We have used this technology to convert classroom assessment techniques (Angelo and Cross ****) to VCATs. Again, as with simple email, the bulk of incoming email may be overwhelming and unmanagable for faculty.

3. Browser forms collected to a database. The responses to a form on a web page can be collected directly to a file on the webserver. This requires that the web-page form calls the appropriate cgi on the webserver. Data from forms can be collected into a text file and then retrieved and sorted or compared using spreadsheet software such as Excel. This allows the faculty to use much more involved questions and collect more information than would be feasible using email directly.

4. Browser forms with automated feedback. A higher level of sophistication is to build cgi scripts that automatically analyze and respond to student input. These cgis require special scripts on the server and each of these scripts may need to be modified and kept current. These types of applications can take a significant amount of time by the faculty or support staff. Constucting these types of applications for several faculty at once is a possibility but you must expect and plan for updating for subsequent terms. This can be supported in many web course management software packages.

5. Pre-enrollment self-assessment quizzes. We have implemented a set of pre-quizzes or prior knowledge probes for large classes. The faculty in the courses determined what skills and knowledge they thought students would need to be successful in their course. Representatitve samples of these skills or content were turned into quiz type questions and placed in web page forms. Some faculty used multiple choice questions that could be automatically scored with an answer returned immediately to the student. Other faculty used short written answers or short paragraph format answers that were collected and analyzed individually by the faculty or teaching assistant. The intent of these pre-quizzes was to allow students an opportunity to asess for themselves, whehter they were ready for the course. The information from these pre-quizzes was collected and analyzed at the beginning of the term and average or aggregated data (with no student names associated with it) was returned to the faculty so that they could use this in readjusting their course for the term.

6. Spreadsheet for grades. Many faculty use a spreadsheet to keep track of student grades in their courses. In addition to tallying the scores, a speadsheet can be used to do simple comparisons between graded assignments and classroom assessments or other learning strategies used in the course. We used a set of inter-related spreadsheets within and Excel workbook to track, classroom assessments and quiz scores on particular questions. Although this seems to be more an analysis function (compared to formative assessment for the student or course), returning that information to students after a quiz may help them focus their subsequent study efforts. For a more complete description of this use of the data, see the chapters 7 and 8.

7. Gradeserver. Another use for spreadsheet gradebook is to make the contents accessible to students through a web page. We implemented this by having faculty send a text version (with line breaks) of the spreadsheet to the webserver. The text file was formated to have the student's last name and a four digit ID number in the first and second columns. The webserver would return the contents of that line, if the name and ID number match. Using a "gradeserver" allows students to get their grades from a quiz or assignment as soon as it is graded. This can provide much quicker feedback for students and has some "assessment value" but the major value is to reduce time in class with students checking their grades with you or verifying that a change in score was actually entered onto the gradesheet. The spreadsheet can be set up to include comments. I have used Excel to automatically generate comments that told students what grade they could expect to earn given their current performance.

8.Checking student records. Many schools are offering electronic access to student records for teaching and advising purposes. This access may be through a web-browser, client-side application or "screen" interface. Faculty can use this information to check on students' background and pre-requisite courses. It may not be possible (or even desirable) to perform a global query on all students for pre-requisite courses and grades, but it may be possible to check on some students. For example, on the first quiz in a large class (> 150 students) six students had difficulty answering even the simplest questions. Checking the student database for these students, five of these students were on academic probation or special admit status indicating that they were not probably well prepared and their poor quiz performance was probably not an indicator of a failure in the course in presenting usable information. This type of information on students is problemmatic for many faculty because they might not to know what to do with the information.

9. Email pagers. We used two-way email pagers to collect student time-on-task data. The study was conducted as part of a larger study. Because of the cost, we were unable to have enough "beepers" to collect information all at once and instead, rotated the beepers five at a time through several sets of students. On a preset schedule between the hours of 7 am and 10 pm the students were beeped and prompted to respond with a code indicating their primary and secondary activity and the percent of their attention devoted to each of these activities. The data was collected in an email file and automatically translated into a database for analysis. Although our study didn't collect enough data to find any significant trends, the technology could be very useful and might be feasible if the price of the pagers and the price per message were to decrease.

Integration of these technologies into a class

The key to the success of employing these technologies is that they add to the value of the assessment exercise for students and faculty or make it more efficient (with the same quality). The assessment needs to be integrated into the course in a way that weaves the content of the question to the themes of the course and that students see both an immediate value of the assessment process and a cummulative benefit of devoting time to these exercises. In our work with faculty, it may take more than several attempts with using individual assessment techniques to get the full value out of them. This is both because it takes certain classroom management and course planning skills to use them effectively, but also that faculty are able to adapt these through experience. In contrast to assessment techniques, individual implementations of particular technologies were often achieved with one attempt and were easily adapted to other situations in the class. The portability and adaptability of the information technology strategies does not seem to carry over to the technology enhanced assessment techniques. Faculty should plan to use the same assessment several times to get the experience and should expect that it might take two or three course attempts to have these techniques comfortably integrated into the course.

In our work with faculty on courses and programs, we have seen many good examples of using technology to support assessment. Below we present four examples that show the range of technology that could be employed - from email to full database applications.

Example 1 - Email and listserv: - Carl Wamser's organic chemisty

Example 2 - VCATs: In General Biology, a forms to webserver assessment technique was routinely used to collect data on what students were having trouble understanding. This course was organized into two week modules. The web-site was extensive, including course notes and other resources. Students were asked to respond to a web page form by the first Friday at 5:00. The form asked them to give a short outline of the content of the reading assignment for that unit and identify one or two points that they thought they could use more help understanding. During the following Monday, the instructor was able to respond to questions that were raised by students, either by giving them hints or addressing them in class.

Example 3 - Checking assumptions on student learning: In a course with a large number of connected specific learning objectives it may be unwieldly to create an comprehensive concept map. Causal analaysis modelling attempts to link prior knowledge with current actions. This approach also provides an underlying model for course and assessment design that lends itself to implementation through databases **** (Jonassen, et al. 1996)**** get this reference ****

Example 4 - Program assessment using an interactive database: As part of a general studies degree that is offered at several regional sites, we developed an interactive database of program goals. The advisory committee for this interdisciplinary program developed broad goals that should be met by students in the program. At the off-campus sites, students have a very limited selection of courses that will lead to completing the degree, therefore the administrators of the program selected courses to meet these goals. The students were taught to understand these goals and be able to judge if they had met these goals as part of an introductory course at the beginning of the program. Students were then asked to identify which of the twenty or so goals were addressed in the courses that they took in a particular term. A web interface was built for the goals database that allowed students to enter a short description of how particular courses contributed to these program goals. None of the courses are expected to meet all of the goals, and some of the courses may only introduce the goal or contribute partially. The faculty were also asked to indentify which goals their course addressed and how it helped students meet each goal. From the students' point of view, they would see a goals and course matrix that lists how they thought each course contributed to goals. Their ultimate goal is to make sure that all program goals have been met either through courses or some other educational activity. The students can also compare how they thought the course contributed to the goals to how their professors thought they helped meet the goals. This assessment effort and student input is expected to last over two years and help keep these students on track and provide one component of the program assessment plan.

Implementation issues and Specific details on technology

TABLE 2. PROJECT STEPS

References

Jonassen, D.H., Mann, E., & Ambruso, D.J. (1996). Causal modeling for structuring case-based learning environments. Intelligent Tutoring Media, 6: 103-112.