2315461код для вставки
A Case Study of a Cooperative Learning Experiment in Artificial Intelligence FERNANDO DÍEZ, RUTH COBOS Departamento de Ingenierı´a Informática, Universidad Auto´noma de Madrid, Francisco Tomás y Valiente 11, Madrid 28049, Spain Received 5 January 2006; accepted 1 November 2006 ABSTRACT: This article describes an innovative teaching experiment (part of a project for Innovation in Teaching at the University Autónoma of Madrid) which was undertaken by the authors during the first semester of the academic year 2004/2005. This teaching experiment has been the object of evaluation by the students as part of their coursework and has consisted of the use of the groupware system KnowCat, by which the students prepare a repository of documents related to topics and themes associated with the subject matter (Artificial Intelligence). During the process of elaboration both the votes for the best documents and the annotation made about them play an essential role. These documents are carried out exclusively by the students and they are who decide, by means of their activity, which of the documents presented are to be chosen as representative of the entire collection. ß 2007 Wiley Periodicals, Inc. Comput Appl Eng Educ 15: 308 316, 2007; Published online in Wiley InterScience (www.interscience.wiley.com); DOI 10.1002/cae.20114 Keywords: cooperative/collaborative learning; evaluation methodologies; knowledge management; undergraduate education; European Credit System; Artificial Intelligence; groupware INTRODUCTION The creation of a joint project related to European Higher Education Space (EHES) presupposes profound changes for Spanish universities in the next few years. The implementation of a new European Correspondence to F. Dı́ez ([email protected]). ß 2007 Wiley Periodicals Inc. 308 Credit System ECTS, and the elaboration of new curriculum and study plans also presuppose important and innovative changes in the totality of degrees and diplomas offered by different State Universities. Each year and in line with these potential changes, the University Autónoma of Madrid—UAM for short—has designed and put into practice Innovative Teaching Projects in an attempt to approximate our current teaching practices to the new EHES requirements. The outcomes of one of these projects are COOPERATIVE LEARNING EXPERIMENT described here below in which we present the results obtained from using a collaborative teaching system as support for Artificial Intelligence Studies in the High Polytechnical School at UAM. This experiment meets the criteria for new and innovative educational content and furthermore in the High Polytechnical School we enjoy a rich learning environment (both in terms of laboratories, technological resources and access facilities) which makes these kinds of ICT educational experiences highly productive. The experiment is part of the Innovative Teaching Project—ITP for short—called ‘‘Active and tutored Learning in the cooperative generation of teaching materials on the Web with the assistance of the KnowCat System,’’ whose principle objectives are as follows: * * * * * To increase and encourage group work among students studying the same subject matter. To generate quality teaching material that is accessible on the web as support for face to face instructional modes. To introduce new methods for evaluating knowledge and other skills acquired by the students. To facilitate active learning in students with the use of the computer. To provide continuous evaluation and tutoring facilitates for students. Based on these objectives, the Artificial Intelligence professors proposed to participate with this subject in the ITP in order to enrich the teaching and to obtain additional work from the students, which will be useful in the task of evaluating them. From our point of view, the work that can be done with the KnowCat system can especially encourage and strengthen cooperative group work outside the traditional classroom which is in line with the general objectives of the new EHES. In addition, from the professors’ perspective the kind of work proposed facilitates active and verbal learning and is in line with the styles of learning proposed by Felder and Silverman . The KnowCat system has been designed to assist not only the student but also the professor in the process of distance learning and teaching . It has been tested previously in other subjects as diverse as ‘‘The Biology of Development,’’ ‘‘Teaching Mathematics to Infants,’’ ‘‘New Information Technology and Applied Education Communication,’’ ‘‘Operative Systems,’’ ‘‘Uncertain Reasoning,’’ ‘‘The Theory of Automatas and Formal Languages,’’ ‘‘Learning Strategies,’’ and ‘‘Psycho Pedagogical Intervention,’’ among others. 309 In the case of the academic subject of interest here, in the next section we will set out the characteristics of the subject as well as the different reasons that we believe justify the development of an experiment like the one carried out. In the third section, we will present the system we employed: KnowCat. In the fourth section we will explain in detail the experiment carried out and in the fifth section the results will be described. Finally, we will present the conclusions we have drawn with respect to the use of the KnowCat system in the field of Artificial Intelligence. THE COURSE SUBJECT: ARTIFICIAL INTELLIGENCE The subject of Artificial Intelligence (http://www. ii.uam.es/esp/alumnos/c3_ia.php) is a fundamental course in the second stage of Computer Science at the High Polytechnical School. In the academic year 2004/2005, this course has three groups of approximately 100 students in each group and is taught through theoretical and practical classes. Each student has 3 h per week of theory and 2 h in the laboratory. The total number of ECTS credits is estimated as 5.6. The students’ work is evaluated at the end of the course by means of a grade based on a final exam, together with a grade for practical work completed and the grade received in a midterm exam. Moreover, this year, 15% of the grade received in the final exam was in fact based on the work done by the students in the ITP. As we mentioned earlier, the subject that is the object of this study is a 2nd stage fundamental course and for the first time in this degree program aspects related to Artificial Intelligence have been included. Later on other subjects will be included that will complete the contents that have been introduced in this subject. The contents of Artificial Intelligence are fundamentally related to search engines (non-heuristic, heuristic, and adversarial) as well as topics related to logic which make up an important part of the content of the course subject. More specifically, the following areas are parts of the subject: propositional logic and predicate calculus. Since this course is an introduction to the subject, the responsible professors believe that it is appropriate to complement the training of the students with their work in ITP. In order to do this we proposed to the students that they study various topics related but not directly linked to the subject area in theoretical, technological or methodological aspects. The following is a list of the topics proposed: 310 DÍEZ AND COBOS (1) History and Perspectives on Artificial Intelligence. (2) Philosophy in Artificial Intelligence. (3) Ethics in Artificial Intelligence. These three proposed topics deal with aspects related to the subject from a historical or moral perspective. It is worth pointing out that one of our initial objectives, and stated here above, was to encourage active learning and generate quality teaching material. One way of achieving both is by motivating students to read texts that go beyond theory and technology, and which invite students to reflect on questions that have to do with the existence of Artificial Machines, the evolution throughout history of the concept of an artificial agent capable of reason and making decisions, what problems have arisen or will arise from the point of view of man’s relationship with machines, the possibility of treating these machines like humans, what obligations might they have and what rights might they have. It is very clear that many of the questions that we have proposed are open questions and almost seem to be Science Fiction. Nevertheless, as we reflect on these issues we are simply dealing with questions that sooner or later modern society in general will have to face. In this way from the very beginning of the course, we have tried to encourage critical thinking and analysis of questions like the above mentioned, to help the students to mature not only with respect to the theoretical aspects presented in the fundamental course, but also with respect to adjacent themes. A DESCRIPTION OF THE SYSTEM USED We have been using the KnowCat system since the academic year 1998/1999 as a teaching support system for various courses in different degree programs at the Universidad Autónoma de Madrid and at the Universidad of Lleida [3 5]. Although the system is used mostly in teaching environments where the primary objective is to generate annotations and high quality teaching material as a result of the interaction of the students with the materials, it could also be used by any community that wished to share general information and knowledge in a distributed way and without the need for supervision. The name of the system, KnowCat, is an acronym for Knowledge Catalyzer (or catalyzer of knowledge) which makes reference to the primary property of the system: ‘‘the catalyzation of the process of knowledge crystallization.’’ We have not included in this article a description of the process since we consider that this does not fall under the objectives of this article. For more information related to this process we encourage readers to refer to [6,7]. The system allows us to create ‘‘Web Spaces’’ or Knowledge Sites where we can find relevant and quality knowledge about a specific area or topic. These Knowledge Sites are called ‘‘KnowCat nodes’’ and they are accessible through the Web by means of a URL. The principal characteristics of the knowledge that we find in a KnowCat node is that it is explicit and stable over time . Furthermore, this information undergoes a process of crystallization due to the fact that we receive constant and consistent feedback with respect to the relevance of the information through the opinions of the users. The knowledge in a KnowCat node is organized in a hierarchy of topics or nodes (each one of these divisions is a KnowCat node, too) which is called a ‘‘tree of knowledge.’’ The root of this tree corresponds to the main topic of the knowledge area in question. In the case of the experiment described here, the name of the root was ‘‘Artificial Intelligence’’ and can be found at the following address: http://knowcat.ii.uam.es/ia/. Each topic contains documents or articles, which are the basic units of information in the system. All the documents contained in a single topic are candidates for describing it. By means of the information crystallization mechanism we are able to tell at any one moment which is the document that best represents the topic and is thus known as the ‘‘crystallized document’’ in this particular topic. The rest of the documents are considered as ‘‘candidates’’ for dominance over current ones and those that are not sufficiently successful over a period of time are later eliminated from the list of the candidate documents. Documents receive explicit opinions by the users who access them. These opinions can be given either by assigning a numerical value, that is the document receives a vote, or by means of an annotation—note for short—which comments or critiques the contents of the document. Votes may be assigned to documents in either of two ways: on the one hand by choosing a value from a range of values and in this way expressing the extent of support given to a particular document, or on the other hand by selecting the unit value and thus expressing support for the document voted. In our experience with students of Artificial Intelligence, most chose the second type for its simplicity. The notes are extremely useful in order to complete the contents of the document in question [3,9]. The notes in the system should only express one single idea and, moreover, they should be classified COOPERATIVE LEARNING EXPERIMENT according to one of the following types: a clarifying note, a support note, a critique, a correction note, an addition note, or a delete note (these last three types are examples of notes for ways to improve the document). Both the explicit opinions received by a document in the form of votes and notes as well as the implicit opinions in the form of number of times the document is accessed, serve to calculate its level or degree of acceptance. Its degree of acceptance and its evolution are used by the crystallization mechanism which determines which documents are representative and as a result are the documents that crystallize. In Figure 1 it is shown an example of a typical screen of KnowCat. The left side of the screen shows the knowledge tree on the knowledge node ‘Artificial Intelligence’. The right side shows the documents that have been added to the topic ‘‘The future: Human, Robot or Cyborg?’’ These documents are identified by their author name, arrival date and a title. They are displayed ordered by their degree of crystallization, which is shown on the right of the identification of each document (with the green-red bar). On the left side of the identification of each document is an icon that informs us if a document has received annotations. The content of a document and its associated annotations are displayed through the selection of the document identification. The KnowCat system has been designed and developed within the framework of the projects known as ‘‘KnowCat: Automatic Catalyzer for 311 Knowledge Crystallization’’ (‘‘KnowCat: Catalizador Automático de la Cristalización del Conocimiento’’) (CAM07T/0027/1998) and ‘‘ARCADIA: Automatic Knowledge Organization, Data Analysis and Dynamic Document Generation in the Semantic Web’’. (‘‘ARCADIA: organización automática del conocimiento, análisis de datos y generación dinámica de documentos en la web semántica’’) (TIC2002-01948). The system is being used as a regular teaching support for various courses thanks to the financial support of the current ITP and by two other Innovation Teaching Projects granted in 2003, one financed by the Universidad Autónoma de Madrid and the other by the Department of Universities, Research and Information Society of Catalonia (Generalitat de Catalunya). THE EXPERIMENT In this section we are going to present a detailed description of the experiment carried out. We have divided this section into three paragraphs in order to provide a chronology of the planning, development and finally the evolution of the experiment. Planning the Experiment An additional objective to those already set out here was to generate a repository of documents only by the students. The contents of this repository were distributed in the mentioned topics above, which are Figure 1 Example screen of the KnowCat system: KnowCat node about Artificial Intelligence. [Color figure can be viewed in the online issue, which is available at www.interscience.wiley.com.] 312 DÍEZ AND COBOS an extension to the content matter covered in class. This experiment was a complete innovation for all the students not only in terms of the elaboration of their own material, which was to be read and assessed by their classmates at the same time, but also in terms of the use of a collaborative work tool. As we mentioned earlier, the repository of documents was completed and evaluated entirely by the students themselves. As the subject professors, we carried out an indirect supervision of the course which was focused on the adequacy of the work with respect to the objectives and guidelines proposed, as well as the correction of the contents of the documents. On the other hand all classification and evaluation of documents was completely done by the students themselves using a system of votes and notes as mentioned above in an earlier section. Development of the Process The 5th Call for participation of Innovation Teaching Projects at UAM had, as part of its fundamental objectives, several initiatives directed at the implantation of the EHES. With this as a general goal, the call for projects invited participants to present projects that included, among others, the following proposals, conditions and criteria (http://www.uam.es/novedades/ innovaciondocente.html): * * * * * * Projects that define, develop and evaluate learning and training programs based on ECTS credits. Projects that encourage and facilitate active learning by students through innovative and pro-active methodology. Projects that involve increased group work; learning based on problem solving, seminars and directed study. Projects that generate new didactic materials that provide support for traditional face-to-face classroom teaching, including materials accessible on the web. Projects that introduce new methods of evaluating information and knowledge and other skills acquired by students. Projects that encourage the application and use of the new technologies and communication to the design, programming and evaluation of curricular materials. The project encompassing this experiment met the requirements of most of these conditions. Once we were given the go ahead we initiated the first phase by establishing the work environment in which the experiment was to be carried out. At the beginning, during the month of September 2004, we verified the working condition of the server where was the KnowCat system, testing and trying out all the necessary technical aspects. In order to do this we had to train a student assistant whose grant had been included in the project proposal. The development of the project in relation to the students was structured into four phases: (i) Student training. (ii) Elaboration and storaging of documents. (iii) Reading, annotating and voting for documents. (iv) Crystallization. Student Training. Once the course had started, at the beginning of October 2004, the three groups of students were told about the existence of the project and the conditions under which is was to be carried out. Firstly, all the students were given a training session where they received instructions about how to use the system. These sessions took place around the middle of October 2004, during which the students were taught the basics: told how to log on to the system, how to edit documents, in which format to work, how to make and manage their notes and votes, etc. Elaboration and Storing of Documents. Once the training seminars were completed the Artificial Intelligence knowledge node was initiated and the students began a month long learning period in which they have to read and analyses information in order to create an own document about one assigned topic from the three proposed topics, and finally to deposit it in the suitable topic. Reading, Annotating, and Voting for Documents. At the end of the month no further documents could be included in the system and thus began the reading phase of the documents deposited, followed by the annotating and voting phase by the students on the documents from their classmates. In order to enrich the documents the students were asked to try to make notes that reflected ways in which the documents could be improved. Each topic contained around eighty documents and so the students were directed to make an effort to give to their documents attractive titles in order to motivate to the rest of their classmates to read his/her document. This second phase concluded at the end of December 2004. The students actively participated in this second phase and the great amount of them complied with the rule that required them to make at least six annotations and three votes to their classmate’s documents. COOPERATIVE LEARNING EXPERIMENT Crystallization. The crystallization process began as soon as the previous phase was completed and in this way we obtained in each topic a classification of the documents from the document with the highest acceptation degree to the document with the lowest acceptation degree according to the students’ opinions. The first document of each classification became the crystallized document. In two of the topics—History and Philosophy—the crystallized or most highly accepted document according to the students’ opinions was also the most representative document according to the professors. However, in the case of the third topic—Ethics—the document in second position was the most highly regarded by the professors. Thus, in this case, an exception had to be made with this topic and the second document was selected as the crystallized document (there was a small difference between the acceptation degree of the first and the second document in this topic). In this way, three quality documents which were representative of the material for the exams had been selected not only by the professors but also by the students. Once the process had been concluded a list was drawn up of the students who were considered eligible to answer the questions in the final exam and receive a grade for the project. Those students who had not completed the tasks were excluded from the process and as a consequence did not receive a grade. Evaluation The final evaluation had as its objective the three crystallized documents mentioned above. The students had to read these three documents. There were three questions constituting the final exam based on these three documents (one for each topic). Each question had assigned a maximum value of 0, 5 points (out of 10) which gave us 15% of the final grade proposed in the course evaluation guidelines pre- 313 sented to the students at the beginning of the semester. The questions formulated were related to the contents of the selected documents. For example, in the case of ‘‘History and Perspectives on Artificial Intelligence,’’ the following question was asked: Any problem that can be solved with the use of a computer should be framed within the field of Artificial Intelligence? Give a rational answer and provide examples to illustrate your answer. The final results of the evaluation of the questions were quite satisfactory and the following average scores were yielded: History 0, 38 points, Philosophy 0, 24 points and Ethics 0, 39 points. RESULTS OBTAINED At the end of the semester the students were asked in an anonymously way to a questionnaire with respect to their work with the system. The results of this questionnaire are shown below. Firstly, we asked them about the number of hours that they had dedicated to each task in the different phases of the experiment. On average they spent 15 h on the task of looking for information and writing their own document on the topic assigned, 9 h reading the work of their classmates, 9 h making notes and annotating and 6 h reading the documents that were chosen as the best, that is the crystallized documents. It was expected that they would dedicate more time to produce their document because their quality was to be taken into account in the final evaluation and classification (cooperatively done) in the system. It is normal that the time it took to read and the time it took to annotate the documents should coincide, given that these two tasks are done at the same time, see Figure 2. Since there were in the order of 80 documents for every topic we did not expect the students to read all Figure 2 Distribution of time spent on tasks. 314 DÍEZ AND COBOS Figure 3 Criteria for choosing documents to read and percentage of numbers of documents read. of them in order for them to assign a value so we asked them what had motivated them to choose the documents to read and how many documents had they read. As can be seen in Figure 3, most of the students chose the documents they would read without any specific criteria (43% of the students), or by the title (28%) or by the name of the author (26 %). The first two options are desirable as it is reasonable to either select without any special criteria or be guided by the title of the document. In fact, giving the document a good title is like assigning it an effective ‘‘business card.’’ With respect to the number of documents read, most of the students read between 6 and 15 documents which is not a low number although it would have been preferable if they had read more documents in order to have a more complete criteria when making their evaluations. Next we asked the students about what had motivated them to vote for or make notes to some documents and not to others, see Figures 4 and 5. In both operations the main criteria that the students used was the quality of the contents of the document (80% of the students voted for the best in their opinion and 36% made annotations for the same reason). Furthermore, it is important to point out that another important reason for making notes was given as ‘‘annotating documents that had important errors’’ (33%). This implies that the students used their notes to both improve and correct the document of their classmates, which was one of the implicit objectives of this experiment. We asked them if they approved of the way in which their document had been evaluated, that is if they were satisfied with the assessment of their document by the rest of the students. The majority of the students agreed with the evaluation of their work (66% of the students polled), however, some of them thought that their document had been overvalued (16%) or undervalued (18%), see Figure 6 below. We asked the students to tell us at what moment they had learned the most from using the system and the majority responded that they benefited most when they read their classmates’ document (37%), in the first phase of reading documents, and when they prepared their own document (35%). It is also important to point out that a large number of students recognised the usefulness of reading the crystallized documents (21%) as well as to make notes to their classmates’ documents (7%), see Figure 7 below. As was expected, not only the preparation and completion of their own documents helped them to comprehend and better understand the contents of the Figure 4 Reasons for voting documents. COOPERATIVE LEARNING EXPERIMENT 315 Figure 5 Reasons for annotating documents. Figure 6 Level of student satisfaction with collaborative evaluation of their documents. topic on which they were working, but also reading and interacting with the work of their classmates was of great assistance to them. CONCLUSIONS AND FUTURE DIRECTIONS Given the results described above, our global assessment of the experiment is a positive one. Fundamentally, this experiment provides evidence that students are predisposed to this type of work, which is an innovation for them, where they are given the responsibility of elaborating acceptable teaching content, overseen not only by the professor supervising the process, but also by each other. The high percentage of students who were satisfied by the evaluation of their assignments (66%) makes us believe that students are motivated by the idea of working with equals, collaborating on the elaboration of content while never losing sight of the competitive nature of the experience. This circumstance that we have described above leads us to believe that our idea that the uses of collaborative tools in a field like Computer Engineering is well received by students. The major difficulties that we encountered came from the need to both Figure 7 Distribution of benefit from working with the system. 316 DÍEZ AND COBOS establish and communicate adequately what was to be the framework in which the experiment was to be developed given that it also included a part of the evaluation. Thus the information, the different phases and the overall development of the experiment had to be very clear from the beginning and leave nothing to chance. As was to be expected, almost all the students’ documents were well developed and written, although since there were many documents for each topic, most of the documents had the same basic contents. Nevertheless, we were pleasantly surprised by some students who produced some quite innovative work that went beyond the normal scope of contents. On the other hand, we should point out that the annotations made are very rich in content and help in the understanding of each document as well as highlighting the distinctive nature of each one. With a view to the new academic year, the project had been extended for one more year, so we are working in the idea of incorporating into the project several novelties, including public debate about the crystallized documents. Each of these face to face debates will include the presence of the professor as moderator and allow the students to compare and contrast ideas pertaining to the topics they will be studying later on and about which they will be elaborating information by means of the KnowCat system. ACKNOWLEDGMENTS The KnowCat system is partially financed by the Ministry of Science and Technology and has the project numbers TIC2002-01948 and TSI200508225-C07-06. The experiment described here has been developed in the framework of the Innovative Teaching Project with a grant from the UAM during the academic year 2004/2005: ‘‘Active and tutored Learning in the cooperative generation of teaching materials on the Web supported by the KnowCat System,’’ with the technical support of Fernando López Colino. REFERENCES  R. M. Felder and L. Silverman, Learning and teaching styles in engineering education, Eng Educ 78 (1988), 674 681.  E. B. Susman, Cooperative learning: A review of factors that increase the effectiveness of cooperative computer-based instruction, J Educ Comput Res 18 (1998), 303 332.  R. Cobos and M. Pifarré, Learning among equals in the Net: Analysis of KnowCat supporting group work, in HCI related papers of Interacción 2004, R. Navarro-Prieto and J. Lorés (Eds.), Springer, Dordrecht, The Netherlands, 2005, pp 281 290.  R. Cobos and X. Alamán, Creating e-books in a distributed and collaborative way, Electron Libr 20 (2002), 288 295.  M. Gómez, A. Gutiérrez, R. Cobos, and X. Alamán, Collaborative Learning with computer-based support in the design of material for the development of abstract thought in Preschool Education. An Experiment in the teaching of mathematics, in Proceedings of the International Symposium of Computers in Education, 2001, 241 254.  X. Alamán and R. Cobos, KnowCat: A Web application for knowledge organization, Lect Notes Comput Sci 1727 (1999), 348 359.  R. Cobos, Mechanisms for the crystallization of knowledge, a proposal using a collaborative system, Doctoral Thesis, Universidad Autónoma de Madrid, Spain, 2003.  V. Allee, The knowledge evolution, Butterworth Heinemann, Boston, 1997.  J. J. Cadiz, A. Gupta, and J. Grudin, Using Web annotations for asynchronous collaboration around documents, Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work, Philadelphia, 2000, pp 309 318. BIOGRAPHIES Fernando Dı́ez is an assistant professor in the Department of Computer Engineering at the Universidad Autónoma de Madrid. He holds a PhD degree in computer science. During the late 1980s and the 1990s he worked in several computer science R&D departments, and his work during this time was related to the development of different types of systems in the AI field. His main interests currently focus on the development of systems to help students learn mathematics and on information-retrieval techniques. More information is available at www.ii.uam.es/fdiez. Ruth Cobos is an assistant professor in the Department of Computer Engineering at the Universidad Autónoma de Madrid. She received her PhD with honors in computer science in 2003. Between 2004 and 2006 she had three postdoctoral stays at the Technische Universität München (Germany), doing research on distributed applications, computer-supported cooperative work, intelligent agents, multi-agent systems, and ontologies in open environments. During the last eight years she has participated in several research projects related to groupware and communityware aimed at distributed knowledge management, education, and learning. More information is available at http://www.ii.uam.es/rcobos.