Unidad de Investigación ACCESO Departament de Psicologia Evolutiva i de l'Educació de la Universitat de València.
Formación, investigación y desarrollo sobre tecnología, educación y discapacidad.

Unidad Acceso > Publicaciones > Esta publicación

Presented to ICCHP 2000. International Conference on Computers Helping People with Special needs.        
Karlsruhe (Germany) 17-21 July 2000.            
http://szswww.ira.uka.de/icchp2000/wgs.htm

Educational Evaluation Supported By Multimedia.
An Assisted Evaluation System

Alcantud, F.; Ferrer, A.M.; Romero, R.; Ordóñez, T; Iriarte, J.M.

Research Unit Acceso http://acceso.uv.es

Dpto. Psicología Evolutiva y de la Educación

Universitat de València Estudi General (Spain)

In this paper we will make a short analysis about the characteristics and procedures that appear in computer assisted evaluation systems. Then we will present SEA (Sistema de Evaluación Asistida), an authoring system in Spanish developed in our University by our Research Unit. SEA produces educational tests that can be administered to students with physical impairments and also to average students. A study showed that SEA does not introduce any significant negative effect on the results of a test when compared to the same test in traditional paper and pencil format for the same group of users.

1. Introduction

In the scientific publications of the last two decades we have found different terms referred to different ways of computerized evaluation (Computer-Based-Assessment). The computerized evaluation includes several technical strategies that have in common the use of the computer for the evaluation of dexterities or ability levels.

In diagnosis and learning contexts it is possible to establish two big lines of computerized research evaluation, starting from the objectives and evaluation procedures followed for each one of them. One of these lines, with bigger specific weight in the psychometric environment, is the one constituted by studies that take advantage of the computer technology and their potential as calculation tool, for a computerized administration of more efficient tests. The second line, characterized by a predominant interest by the study of the cognitive processes and its implication in the realization of tasks, takes advantage of the characteristics of the computer when carrying out dynamic methods of evaluation.

For SEA development we had to analyze the characteristics and procedures used in the systems of tests administration. This research was necessary for the development of a system that allows to administer psychological or educational tests to students with physical disabilities.

1.1. Computerized Tests.

Computerized tests (Computer Based Examination or Computer Administered Test.) include paper and pencil tests directly transfered to computer support, as well as tests that are designed in their origin for the computer, inside the traditional correction application.

Among the developments carried out in the last years in this area, we can mention the WICAT Systems battery [11], for the evaluation of diverse basic skills by means of visual and auditory stimuli; the program LAMP [11] that measures different procedural variables applied to the verbal, space and quantitative domain and the TASKOMAT battery [11] that includes several measures of time of reaction before visual stimuli, short term memory, visual selective attention, auditory selective attention and dynamic perception.

We can also mention some authoring tools programs like ASC Software Information MicroCAT [6],[8], Question Mark, [5] C-Quest [2], Examiner [3], FastTEST [4] or FastTEST Pro [7]. The program Test Pilot [1] would also enter in this category, as it is specially designed for the administration and tests correction through the net.

1.2. Computerized Adaptative Testing.

Computerized Adaptative Testing are characterized mainly by their adapting capacity to the user's answers. The theoretical advances in Psychometrics, mainly in connection with the models of the Theory of Answer to the Item (TRI), allow a new type of computerized evaluation that it is also adaptative, since the presentation of a new item depends on the answers carried out from the previous items, allowing a more precise estimation of the evaluated aspect.

Nevertheless, in spite of the many possibilities that provide the TRI to generate new evaluation types and their psycometrical advantages, there are few practical applications that have been developed even though they are technically feasible. It is a research area that is only beginning to mature.

Nowadays, among the available computer programs to carry out computerized adaptative evaluations, we can mention: PETA [10]; FastTest Pro [7] and MICROCAT [6],[8]. In Spain there are pioneer applications such as DEMOTAC [14], METRIX Engine [14] and ADTEST [12],[13].

2. Adapted evaluation for people with disability. SEA program

The evaluation of students with physical or sensorial disability has always been an unresearched subject. Among the systems of computerized evaluation we have not found any reference to the use of these instruments with the purpose of compensating the differences that a disability introduces in the standard evaluation. This communication presents the system S.E.A., a program designed for the development of accessible evaluation applications for students with physical disability. This system allows the psychologist, professor, or another education professional, to develop computer applications in which different items are presented in different windows structured in learning units. Variables can be defined for the system and their value can be taken into account automatically during the evaluation to alter the flow of the program and its actions. The student will be able to select the alternative wanted by means of the keyboard, an ordinary or adapted mouse, or by means of a simple switch. The user answers are registered by the system for later analysis and evaluation.

2.1. Edition interface

 In the figure 1 it can be seen the main screen of the interface designed for the Assistive Evaluation System (SEA). In the menu bar the File and Edit commands correspond to the usual functions of any Windows program.

The system is designed so that the professional can edit the evaluation test, considering that each item occupies a complete presentation in screen and determining the presentation sequence and the system of appropriate answer for the student and the test type.

Next to the conventional menus of Windows applications, specific menus of the program appear. These are Object, Item, Options, Variables and Evaluations. In this paper we will present some of the most interesting options.

Figure 1. Main application screen

An evaluation item is a collection of  objects which are presented simultaneously to the user in the same screen. Items are usually presented to the user sequentially for him to make a choice between them according to a test question. Once the application test is created, it is possible edit,  insert, eliminate or alter the order of the items.

When an application is active, the area of the item occupies the complete screen of the computer, the tools bar and the menu bar disappear and, if it has been specified in the options of the environment to answer with switch, a sequential process of scanning begins among the objects included in the item.

2.2. Evaluation process.

The selection of an evaluation model and the definition of the variables to be considered in the test are carried out from the Variables menu. When this menu is clicked, the following sub-menus are available (see figure 2):

1. Evaluation variables

2. Options

3. Internal variables

Figure 2. List of variables by default

These variables reflect the evolution of the student in the test: correct and wrong answers, total time used and mean time per item, and so on.

In an advance mode it is possible to define numerical expressions based in the previous variables that allow to the system to take decisions about the test execution. For example: “If Total_wrong_question > 5 then finish test”.

Lastly, once the evaluation is finished, it is possible to get a report of the questions that the student has answered and the path followed until arriving at the end of the test.

3. Conclusions

Currently (April 2000) we are finishing the development of SEA and we are verifying the validity of the administration. We have developed different evaluation tests, some of psychological character others of educational character. We have applied to a students' sample in an experimental study, the test of Columbia and the another test. We have used the traditional paper and pencil format and also administered the test through the computer using SEA to all students. We have counterbalanced the application order, so that half of students started with SEA test and the other half with the traditional format.

The correlation among the two application forms was higher than 0.80. In consequence, we can conclude that the test administration by means of this computerized system does not introduce any significant negative effect on the results and hence it allows to include students with physical disability.

A web site has been developed to support the implementation of the system: http://acceso3.uv.es/sea. SEA is freeware and can be downloaded from this site. The site offers also the latest news about the program, updates, sample applications and patches and a link to the developers forum created to offer the possibility of sharing information and tips for the program. All the site and forum are in Spanish language.

4. Acknowledgements

SEA has been developed during the projects UNICHANCE and ACCESO25 of the Horizon Initiative of the European Social Fund with the co-financing of Universitat de València Estudi General.

5. References

[1]     ASC SOFTWARE INFORMATION (1988) IRT Analysis Software. Assessment Systems Corporation University Ave., Suite 200 St. Paul, MN 55114 Estados Unidos. http://www.assess.com/softsum4.html

[2]     ASC SOFTWARE INFORMATION (1988) C-Quest Test Authoring and Administration. Assessment Systems Corporation University Ave., Suite 200 St. Paul, MN 55114 Estados Unidos. http://www.assess.com/C-Quest.html

[3]     ASC SOFTWARE INFORMATION (1988) The Examiner Testing System. Assessment Systems Corporation University Ave., Suite 200 St. Paul, MN 55114 Estados Unidos. http://www.assess.com/examiner.html

[4]     ASC SOFTWARE INFORMATION (1988) FastTEST Item Banking and On-Line Testing. Assessment Systems Corporation University Ave., Suite 200 St. Paul, MN 55114 Estados Unidos. http://www.assess.com/fasttest.html

[5]     ASC SOFTWARE INFORMATION (1988) Question Mark Testing System. . Assessment Systems Corporation University Ave., Suite 200 St. Paul, MN 55114 Estados Unidos. http://www.assess.com/qmark.html

[6]     ASC SOFTWARE INFORMATION (1988) MicroCAT Computerized Testing System. Assessment Systems Corporation University Ave., Suite 200 St. Paul, MN 55114 Estados Unidos. http://www.assess.com/microcat.html

[7]     ASC SOFTWARE INFORMATION (1988) FastTEST Professional Computerized Testing System. Assessment Systems Corporation University Ave., Suite 200 St. Paul, MN 55114 Estados Unidos. http://www.assess.com/FastTEST%20Pro.html

[8]     Assessment System Corporation (1994) User's manual for the MicroCAT testing system. (2ª Ed.) St. Paul. Minessotta

[9]     DUNCAN, W. MALCOLM (1988) The Perfect Match. BioMedia Center for Instructional Computing. http://biomedia.bio.purdue.edu/PerfectMatch/

[10]   NITKO, A. J. y HSU, T. (1984) A comprehensive Microcomputer System for Classroom Testing. Journal of Educational Measurement. Vol 21. Nº 4, pp. 337-390.

[11]   ORDÓÑEZ RUBIO, T. (2000) Diseño Y Evaluación De Un Sistema De Administración De Tests Por Ordenador. (Sea: Sistema De Evaluación Asistida). Universitat de València Estudi General. Tesis Doctoral.

[12]   PONSODA, G.; OLEA, J. y ALCÁZAR (1994) APT-SYSTEM: Procedimientos para la evaluación psicológica adaptativa y autoadaptativa. Capital Humano, Nº 65, pp. 28-34.

[13]   PONSODA, V.; OLEA, J. y REVUELTA, J. (1994) ADTEST: A computer-adaptive test based on the maximum information principle. Educational and Psychological Measurement, Vol. 54, Nº 3, pp. 680-686

[14]   RENOM, J. (1993) Tests Adaptativos Computerizados. Fundamentos y Aplicaciones. Barcelona. Promociones y Publicaciones Universitarias, S.A.