вход по аккаунту



код для вставки
A Case Study of a
Cooperative Learning
Experiment in Artificial
Departamento de Ingenierı´a Informática, Universidad Auto´noma de Madrid, Francisco Tomás y Valiente 11,
Madrid 28049, Spain
Received 5 January 2006; accepted 1 November 2006
ABSTRACT: This article describes an innovative teaching experiment (part of a project for
Innovation in Teaching at the University Autónoma of Madrid) which was undertaken by the
authors during the first semester of the academic year 2004/2005. This teaching experiment has
been the object of evaluation by the students as part of their coursework and has consisted of
the use of the groupware system KnowCat, by which the students prepare a repository of
documents related to topics and themes associated with the subject matter (Artificial
Intelligence). During the process of elaboration both the votes for the best documents and the
annotation made about them play an essential role. These documents are carried out
exclusively by the students and they are who decide, by means of their activity, which of the
documents presented are to be chosen as representative of the entire collection. ß 2007 Wiley
Periodicals, Inc. Comput Appl Eng Educ 15: 308 316, 2007; Published online in Wiley InterScience
(; DOI 10.1002/cae.20114
Keywords: cooperative/collaborative learning; evaluation methodologies; knowledge management; undergraduate education; European Credit System; Artificial Intelligence; groupware
The creation of a joint project related to European
Higher Education Space (EHES) presupposes profound changes for Spanish universities in the next
few years. The implementation of a new European
Correspondence to F. Dı́ez ([email protected]).
ß 2007 Wiley Periodicals Inc.
Credit System ECTS, and the elaboration of new
curriculum and study plans also presuppose important
and innovative changes in the totality of degrees and
diplomas offered by different State Universities.
Each year and in line with these potential changes,
the University Autónoma of Madrid—UAM for
short—has designed and put into practice Innovative
Teaching Projects in an attempt to approximate our
current teaching practices to the new EHES requirements. The outcomes of one of these projects are
described here below in which we present the results
obtained from using a collaborative teaching system
as support for Artificial Intelligence Studies in the
High Polytechnical School at UAM. This experiment
meets the criteria for new and innovative educational
content and furthermore in the High Polytechnical
School we enjoy a rich learning environment (both in
terms of laboratories, technological resources and
access facilities) which makes these kinds of ICT
educational experiences highly productive.
The experiment is part of the Innovative Teaching
Project—ITP for short—called ‘‘Active and tutored
Learning in the cooperative generation of teaching
materials on the Web with the assistance of the
KnowCat System,’’ whose principle objectives are as
To increase and encourage group work among
students studying the same subject matter.
To generate quality teaching material that is
accessible on the web as support for face to face
instructional modes.
To introduce new methods for evaluating knowledge and other skills acquired by the students.
To facilitate active learning in students with the
use of the computer.
To provide continuous evaluation and tutoring
facilitates for students.
Based on these objectives, the Artificial Intelligence professors proposed to participate with this
subject in the ITP in order to enrich the teaching and
to obtain additional work from the students, which
will be useful in the task of evaluating them. From
our point of view, the work that can be done with
the KnowCat system can especially encourage and
strengthen cooperative group work outside the traditional classroom which is in line with the general
objectives of the new EHES. In addition, from the
professors’ perspective the kind of work proposed
facilitates active and verbal learning and is in line with
the styles of learning proposed by Felder and Silverman [1].
The KnowCat system has been designed to assist
not only the student but also the professor in the
process of distance learning and teaching [2]. It
has been tested previously in other subjects as
diverse as ‘‘The Biology of Development,’’ ‘‘Teaching
Mathematics to Infants,’’ ‘‘New Information Technology and Applied Education Communication,’’
‘‘Operative Systems,’’ ‘‘Uncertain Reasoning,’’ ‘‘The
Theory of Automatas and Formal Languages,’’
‘‘Learning Strategies,’’ and ‘‘Psycho Pedagogical
Intervention,’’ among others.
In the case of the academic subject of interest
here, in the next section we will set out the characteristics of the subject as well as the different reasons that
we believe justify the development of an experiment
like the one carried out. In the third section, we will
present the system we employed: KnowCat. In the
fourth section we will explain in detail the experiment
carried out and in the fifth section the results will be
described. Finally, we will present the conclusions we
have drawn with respect to the use of the KnowCat
system in the field of Artificial Intelligence.
The subject of Artificial Intelligence (http://www. is a fundamental
course in the second stage of Computer Science at
the High Polytechnical School. In the academic year
2004/2005, this course has three groups of approximately 100 students in each group and is taught
through theoretical and practical classes. Each student
has 3 h per week of theory and 2 h in the laboratory.
The total number of ECTS credits is estimated as
5.6. The students’ work is evaluated at the end of the
course by means of a grade based on a final exam,
together with a grade for practical work completed
and the grade received in a midterm exam. Moreover,
this year, 15% of the grade received in the final exam
was in fact based on the work done by the students in
the ITP.
As we mentioned earlier, the subject that is the
object of this study is a 2nd stage fundamental course
and for the first time in this degree program aspects
related to Artificial Intelligence have been included.
Later on other subjects will be included that will
complete the contents that have been introduced in
this subject.
The contents of Artificial Intelligence are fundamentally related to search engines (non-heuristic,
heuristic, and adversarial) as well as topics related to
logic which make up an important part of the content
of the course subject. More specifically, the following
areas are parts of the subject: propositional logic and
predicate calculus. Since this course is an introduction
to the subject, the responsible professors believe that
it is appropriate to complement the training of
the students with their work in ITP. In order to do
this we proposed to the students that they study
various topics related but not directly linked to the
subject area in theoretical, technological or methodological aspects. The following is a list of the topics
(1) History and Perspectives on Artificial Intelligence.
(2) Philosophy in Artificial Intelligence.
(3) Ethics in Artificial Intelligence.
These three proposed topics deal with aspects
related to the subject from a historical or moral
perspective. It is worth pointing out that one of our
initial objectives, and stated here above, was to
encourage active learning and generate quality teaching material. One way of achieving both is by motivating students to read texts that go beyond theory and
technology, and which invite students to reflect on
questions that have to do with the existence of
Artificial Machines, the evolution throughout history
of the concept of an artificial agent capable of reason
and making decisions, what problems have arisen or
will arise from the point of view of man’s relationship
with machines, the possibility of treating these
machines like humans, what obligations might they
have and what rights might they have. It is very clear
that many of the questions that we have proposed are
open questions and almost seem to be Science Fiction.
Nevertheless, as we reflect on these issues we are
simply dealing with questions that sooner or later
modern society in general will have to face. In this
way from the very beginning of the course, we have
tried to encourage critical thinking and analysis
of questions like the above mentioned, to help the
students to mature not only with respect to the
theoretical aspects presented in the fundamental
course, but also with respect to adjacent themes.
We have been using the KnowCat system since the
academic year 1998/1999 as a teaching support
system for various courses in different degree
programs at the Universidad Autónoma de Madrid
and at the Universidad of Lleida [3 5].
Although the system is used mostly in teaching
environments where the primary objective is to
generate annotations and high quality teaching material as a result of the interaction of the students with
the materials, it could also be used by any community
that wished to share general information and knowledge in a distributed way and without the need for
The name of the system, KnowCat, is an acronym
for Knowledge Catalyzer (or catalyzer of knowledge)
which makes reference to the primary property of the
system: ‘‘the catalyzation of the process of knowledge
crystallization.’’ We have not included in this article a
description of the process since we consider that this
does not fall under the objectives of this article. For
more information related to this process we encourage
readers to refer to [6,7]. The system allows us to create
‘‘Web Spaces’’ or Knowledge Sites where we can find
relevant and quality knowledge about a specific area
or topic. These Knowledge Sites are called ‘‘KnowCat
nodes’’ and they are accessible through the Web by
means of a URL.
The principal characteristics of the knowledge
that we find in a KnowCat node is that it is explicit and
stable over time [8]. Furthermore, this information
undergoes a process of crystallization due to the fact
that we receive constant and consistent feedback with
respect to the relevance of the information through the
opinions of the users.
The knowledge in a KnowCat node is organized
in a hierarchy of topics or nodes (each one of these
divisions is a KnowCat node, too) which is called a
‘‘tree of knowledge.’’ The root of this tree corresponds
to the main topic of the knowledge area in question. In
the case of the experiment described here, the name of
the root was ‘‘Artificial Intelligence’’ and can be found
at the following address:
Each topic contains documents or articles, which
are the basic units of information in the system. All
the documents contained in a single topic are candidates for describing it. By means of the information
crystallization mechanism we are able to tell at
any one moment which is the document that best
represents the topic and is thus known as the ‘‘crystallized document’’ in this particular topic. The rest of
the documents are considered as ‘‘candidates’’ for
dominance over current ones and those that are not
sufficiently successful over a period of time are later
eliminated from the list of the candidate documents.
Documents receive explicit opinions by the users
who access them. These opinions can be given either
by assigning a numerical value, that is the document
receives a vote, or by means of an annotation—note
for short—which comments or critiques the contents
of the document.
Votes may be assigned to documents in either of
two ways: on the one hand by choosing a value from a
range of values and in this way expressing the extent
of support given to a particular document, or on the
other hand by selecting the unit value and thus
expressing support for the document voted. In our
experience with students of Artificial Intelligence,
most chose the second type for its simplicity.
The notes are extremely useful in order to
complete the contents of the document in question
[3,9]. The notes in the system should only express one
single idea and, moreover, they should be classified
according to one of the following types: a clarifying
note, a support note, a critique, a correction note, an
addition note, or a delete note (these last three types
are examples of notes for ways to improve the
Both the explicit opinions received by a document in the form of votes and notes as well as the
implicit opinions in the form of number of times the
document is accessed, serve to calculate its level or
degree of acceptance. Its degree of acceptance and its
evolution are used by the crystallization mechanism
which determines which documents are representative
and as a result are the documents that crystallize. In
Figure 1 it is shown an example of a typical screen of
KnowCat. The left side of the screen shows the
knowledge tree on the knowledge node ‘Artificial
Intelligence’. The right side shows the documents that
have been added to the topic ‘‘The future: Human,
Robot or Cyborg?’’ These documents are identified by
their author name, arrival date and a title. They are
displayed ordered by their degree of crystallization,
which is shown on the right of the identification of
each document (with the green-red bar). On the left
side of the identification of each document is an icon
that informs us if a document has received annotations. The content of a document and its associated
annotations are displayed through the selection of the
document identification.
The KnowCat system has been designed and
developed within the framework of the projects
known as ‘‘KnowCat: Automatic Catalyzer for
Knowledge Crystallization’’ (‘‘KnowCat: Catalizador
Automático de la Cristalización del Conocimiento’’)
(CAM07T/0027/1998) and ‘‘ARCADIA: Automatic
Knowledge Organization, Data Analysis and Dynamic
Document Generation in the Semantic Web’’. (‘‘ARCADIA: organización automática del conocimiento, análisis de datos y generación dinámica de documentos
en la web semántica’’) (TIC2002-01948). The system
is being used as a regular teaching support for various
courses thanks to the financial support of the current
ITP and by two other Innovation Teaching Projects
granted in 2003, one financed by the Universidad
Autónoma de Madrid and the other by the Department
of Universities, Research and Information Society of
Catalonia (Generalitat de Catalunya).
In this section we are going to present a detailed
description of the experiment carried out. We have
divided this section into three paragraphs in order to
provide a chronology of the planning, development
and finally the evolution of the experiment.
Planning the Experiment
An additional objective to those already set out here
was to generate a repository of documents only by
the students. The contents of this repository were
distributed in the mentioned topics above, which are
Figure 1 Example screen of the KnowCat system: KnowCat node about Artificial Intelligence.
[Color figure can be viewed in the online issue, which is available at]
an extension to the content matter covered in class.
This experiment was a complete innovation for all the
students not only in terms of the elaboration of their
own material, which was to be read and assessed by
their classmates at the same time, but also in terms of
the use of a collaborative work tool.
As we mentioned earlier, the repository of
documents was completed and evaluated entirely by
the students themselves. As the subject professors, we
carried out an indirect supervision of the course which
was focused on the adequacy of the work with respect
to the objectives and guidelines proposed, as well
as the correction of the contents of the documents. On
the other hand all classification and evaluation of
documents was completely done by the students
themselves using a system of votes and notes as
mentioned above in an earlier section.
Development of the Process
The 5th Call for participation of Innovation Teaching
Projects at UAM had, as part of its fundamental
objectives, several initiatives directed at the implantation of the EHES. With this as a general goal, the
call for projects invited participants to present projects
that included, among others, the following proposals,
conditions and criteria (
Projects that define, develop and evaluate learning and training programs based on ECTS
Projects that encourage and facilitate active
learning by students through innovative and
pro-active methodology.
Projects that involve increased group work;
learning based on problem solving, seminars
and directed study.
Projects that generate new didactic materials that
provide support for traditional face-to-face classroom teaching, including materials accessible on
the web.
Projects that introduce new methods of evaluating information and knowledge and other skills
acquired by students.
Projects that encourage the application and use
of the new technologies and communication to
the design, programming and evaluation of
curricular materials.
The project encompassing this experiment met
the requirements of most of these conditions. Once we
were given the go ahead we initiated the first phase by
establishing the work environment in which the experiment was to be carried out. At the beginning, during
the month of September 2004, we verified the working
condition of the server where was the KnowCat
system, testing and trying out all the necessary technical aspects. In order to do this we had to train a
student assistant whose grant had been included in the
project proposal. The development of the project in
relation to the students was structured into four
(i) Student training.
(ii) Elaboration and storaging of documents.
(iii) Reading, annotating and voting for documents.
(iv) Crystallization.
Student Training. Once the course had started, at the
beginning of October 2004, the three groups of
students were told about the existence of the project
and the conditions under which is was to be carried
out. Firstly, all the students were given a training
session where they received instructions about how to
use the system. These sessions took place around the
middle of October 2004, during which the students
were taught the basics: told how to log on to the system,
how to edit documents, in which format to work, how to
make and manage their notes and votes, etc.
Elaboration and Storing of Documents. Once the
training seminars were completed the Artificial
Intelligence knowledge node was initiated and the
students began a month long learning period in which
they have to read and analyses information in order to
create an own document about one assigned topic
from the three proposed topics, and finally to deposit it
in the suitable topic.
Reading, Annotating, and Voting for Documents. At
the end of the month no further documents could be
included in the system and thus began the reading
phase of the documents deposited, followed by the
annotating and voting phase by the students on the
documents from their classmates. In order to enrich
the documents the students were asked to try to make
notes that reflected ways in which the documents
could be improved. Each topic contained around
eighty documents and so the students were directed to
make an effort to give to their documents attractive
titles in order to motivate to the rest of their classmates
to read his/her document. This second phase concluded
at the end of December 2004. The students actively
participated in this second phase and the great amount
of them complied with the rule that required them to
make at least six annotations and three votes to their
classmate’s documents.
Crystallization. The crystallization process began as
soon as the previous phase was completed and in this
way we obtained in each topic a classification of
the documents from the document with the highest
acceptation degree to the document with the lowest
acceptation degree according to the students’
opinions. The first document of each classification
became the crystallized document. In two of the
topics—History and Philosophy—the crystallized or
most highly accepted document according to the
students’ opinions was also the most representative
document according to the professors. However, in the
case of the third topic—Ethics—the document
in second position was the most highly regarded by
the professors. Thus, in this case, an exception had to
be made with this topic and the second document was
selected as the crystallized document (there was a
small difference between the acceptation degree of
the first and the second document in this topic). In
this way, three quality documents which were
representative of the material for the exams had
been selected not only by the professors but also by
the students. Once the process had been concluded a
list was drawn up of the students who were considered
eligible to answer the questions in the final exam and
receive a grade for the project. Those students who
had not completed the tasks were excluded from the
process and as a consequence did not receive a grade.
The final evaluation had as its objective the three
crystallized documents mentioned above. The students had to read these three documents. There were
three questions constituting the final exam based on
these three documents (one for each topic). Each
question had assigned a maximum value of 0, 5 points
(out of 10) which gave us 15% of the final grade
proposed in the course evaluation guidelines pre-
sented to the students at the beginning of the semester.
The questions formulated were related to the contents
of the selected documents. For example, in the case of
‘‘History and Perspectives on Artificial Intelligence,’’
the following question was asked:
Any problem that can be solved with the use of a
computer should be framed within the field of
Artificial Intelligence? Give a rational answer and
provide examples to illustrate your answer.
The final results of the evaluation of the questions
were quite satisfactory and the following average
scores were yielded: History 0, 38 points, Philosophy
0, 24 points and Ethics 0, 39 points.
At the end of the semester the students were asked in
an anonymously way to a questionnaire with respect
to their work with the system. The results of this
questionnaire are shown below.
Firstly, we asked them about the number of hours
that they had dedicated to each task in the different
phases of the experiment. On average they spent 15 h
on the task of looking for information and writing
their own document on the topic assigned, 9 h reading
the work of their classmates, 9 h making notes and
annotating and 6 h reading the documents that were
chosen as the best, that is the crystallized documents.
It was expected that they would dedicate more time to
produce their document because their quality was to
be taken into account in the final evaluation and
classification (cooperatively done) in the system. It is
normal that the time it took to read and the time it took
to annotate the documents should coincide, given that
these two tasks are done at the same time, see
Figure 2.
Since there were in the order of 80 documents for
every topic we did not expect the students to read all
Figure 2 Distribution of time spent on tasks.
Figure 3 Criteria for choosing documents to read and percentage of numbers of documents read.
of them in order for them to assign a value so we asked
them what had motivated them to choose the documents to read and how many documents had they
read. As can be seen in Figure 3, most of the students
chose the documents they would read without any
specific criteria (43% of the students), or by the title
(28%) or by the name of the author (26 %). The first
two options are desirable as it is reasonable to either
select without any special criteria or be guided by the
title of the document. In fact, giving the document a
good title is like assigning it an effective ‘‘business
card.’’ With respect to the number of documents read,
most of the students read between 6 and 15 documents
which is not a low number although it would have
been preferable if they had read more documents in
order to have a more complete criteria when making
their evaluations.
Next we asked the students about what had
motivated them to vote for or make notes to some
documents and not to others, see Figures 4 and 5. In
both operations the main criteria that the students used
was the quality of the contents of the document (80%
of the students voted for the best in their opinion and
36% made annotations for the same reason). Furthermore, it is important to point out that another
important reason for making notes was given as
‘‘annotating documents that had important errors’’
(33%). This implies that the students used their notes
to both improve and correct the document of their
classmates, which was one of the implicit objectives
of this experiment.
We asked them if they approved of the way in
which their document had been evaluated, that is if
they were satisfied with the assessment of their
document by the rest of the students. The majority
of the students agreed with the evaluation of their
work (66% of the students polled), however, some of
them thought that their document had been overvalued
(16%) or undervalued (18%), see Figure 6 below.
We asked the students to tell us at what moment
they had learned the most from using the system and
the majority responded that they benefited most when
they read their classmates’ document (37%), in the
first phase of reading documents, and when they
prepared their own document (35%). It is also important to point out that a large number of students
recognised the usefulness of reading the crystallized
documents (21%) as well as to make notes to their
classmates’ documents (7%), see Figure 7 below.
As was expected, not only the preparation and
completion of their own documents helped them to
comprehend and better understand the contents of the
Figure 4 Reasons for voting documents.
Figure 5 Reasons for annotating documents.
Figure 6 Level of student satisfaction with collaborative evaluation of their documents.
topic on which they were working, but also reading
and interacting with the work of their classmates was
of great assistance to them.
Given the results described above, our global assessment of the experiment is a positive one. Fundamentally, this experiment provides evidence that students
are predisposed to this type of work, which is an
innovation for them, where they are given the responsibility of elaborating acceptable teaching content,
overseen not only by the professor supervising the
process, but also by each other. The high percentage
of students who were satisfied by the evaluation
of their assignments (66%) makes us believe that
students are motivated by the idea of working with
equals, collaborating on the elaboration of content
while never losing sight of the competitive nature of
the experience.
This circumstance that we have described above
leads us to believe that our idea that the uses of
collaborative tools in a field like Computer Engineering is well received by students. The major difficulties
that we encountered came from the need to both
Figure 7 Distribution of benefit from working with the system.
establish and communicate adequately what was to be
the framework in which the experiment was to be developed given that it also included a part of the evaluation.
Thus the information, the different phases and the
overall development of the experiment had to be very
clear from the beginning and leave nothing to chance.
As was to be expected, almost all the students’
documents were well developed and written, although
since there were many documents for each topic, most
of the documents had the same basic contents.
Nevertheless, we were pleasantly surprised by some
students who produced some quite innovative work
that went beyond the normal scope of contents. On the
other hand, we should point out that the annotations
made are very rich in content and help in the understanding of each document as well as highlighting the
distinctive nature of each one.
With a view to the new academic year, the project
had been extended for one more year, so we are working
in the idea of incorporating into the project several
novelties, including public debate about the crystallized
documents. Each of these face to face debates will
include the presence of the professor as moderator and
allow the students to compare and contrast ideas
pertaining to the topics they will be studying later on
and about which they will be elaborating information by
means of the KnowCat system.
The KnowCat system is partially financed by
the Ministry of Science and Technology and has
the project numbers TIC2002-01948 and TSI200508225-C07-06. The experiment described here has
been developed in the framework of the Innovative
Teaching Project with a grant from the UAM during
the academic year 2004/2005: ‘‘Active and tutored
Learning in the cooperative generation of teaching
materials on the Web supported by the KnowCat
System,’’ with the technical support of Fernando
López Colino.
[1] R. M. Felder and L. Silverman, Learning and teaching
styles in engineering education, Eng Educ 78 (1988),
674 681.
[2] E. B. Susman, Cooperative learning: A review of
factors that increase the effectiveness of cooperative
computer-based instruction, J Educ Comput Res 18
(1998), 303 332.
[3] R. Cobos and M. Pifarré, Learning among equals
in the Net: Analysis of KnowCat supporting group
work, in HCI related papers of Interacción 2004,
R. Navarro-Prieto and J. Lorés (Eds.), Springer,
Dordrecht, The Netherlands, 2005, pp 281 290.
[4] R. Cobos and X. Alamán, Creating e-books in a
distributed and collaborative way, Electron Libr
20 (2002), 288 295.
[5] M. Gómez, A. Gutiérrez, R. Cobos, and X. Alamán,
Collaborative Learning with computer-based support
in the design of material for the development of
abstract thought in Preschool Education. An Experiment in the teaching of mathematics, in Proceedings of
the International Symposium of Computers in Education, 2001, 241 254.
[6] X. Alamán and R. Cobos, KnowCat: A Web application for knowledge organization, Lect Notes Comput
Sci 1727 (1999), 348 359.
[7] R. Cobos, Mechanisms for the crystallization of
knowledge, a proposal using a collaborative system,
Doctoral Thesis, Universidad Autónoma de Madrid,
Spain, 2003.
[8] V. Allee, The knowledge evolution, Butterworth
Heinemann, Boston, 1997.
[9] J. J. Cadiz, A. Gupta, and J. Grudin, Using Web
annotations for asynchronous collaboration around
documents, Proceedings of the 2000 ACM Conference
on Computer Supported Cooperative Work, Philadelphia, 2000, pp 309 318.
Fernando Dı́ez is an assistant professor in
the Department of Computer Engineering at
the Universidad Autónoma de Madrid. He
holds a PhD degree in computer science.
During the late 1980s and the 1990s he
worked in several computer science R&D
departments, and his work during this time
was related to the development of different
types of systems in the AI field. His main
interests currently focus on the development of systems to help
students learn mathematics and on information-retrieval techniques.
More information is available at
Ruth Cobos is an assistant professor in the
Department of Computer Engineering at the
Universidad Autónoma de Madrid. She received her PhD with honors in computer science
in 2003. Between 2004 and 2006 she had
three postdoctoral stays at the Technische
Universität München (Germany), doing
research on distributed applications, computer-supported cooperative work, intelligent
agents, multi-agent systems, and ontologies in open environments.
During the last eight years she has participated in several research
projects related to groupware and communityware aimed at
distributed knowledge management, education, and learning. More
information is available at
Без категории
Размер файла
216 Кб
Пожаловаться на содержимое документа