USE OF VOICE TO TEXT RECOGNITION SYSTEMS
FOR AUTOMATIC SUBTITLING OF
SCHOOL LESSONS FOR THE DEAF
AND OF LANGUAGES LESSONS FOR ALL STUDENTS:
PEDAGOGIC ASPECTS AND TELE-EDUCATION SUPPORT
BY A WEB SITE AND AN INTERNET DISCUSSION FORUM
European Commission Joint Research Centre
Institute for Systems, Informatics and Safety
in collaboration with Schools and Associations of the Deaf
The Netd@ys VOICE Project
Summary
The difficulties of the deaf go beyond the loss of hearing itself, and underline a more general problem of lack of communication. Help in reducing the gap between the deaf and the hearing world should be enforced. Automatic recognition of speech in conferences and school lessons, with their translation into PC screen messages, could be a powerful help for the deaf. Also other users could benefit of this technology: people with communication problems, foreigners or immigrated persons studying the host country's language or any person studying foreign languages.
The VOICE Project proposes not only the promotion of new technologies in the field of voice to text recognition, but also to stimulate and increase the use of new, widely diffused technologies, namely the Internet. The objective of the Project is that of uniting, by means of an Internet VOICE Forum, schools, public administrations, services providers and other organisations, who are interested in this research. A support will be offered by multimedia tele-education systems, based on a Discussion Forum and a Chat Line. A Netd@ys Help Desk and a Workshop will help in the initiation and training of teachers to the aspects of disability and more generally of the use of the Internet.
Presentations at schools, a videoconference linked to a television transmission, a world wide Web search will draw the attention of the public on these themes. The final goal is broader than the technical aspects related to speech recognition. The aim is that of a large awareness campaign on the problems encounter by students with special needs in the school and in the society, in everyday aspects of their life. The use of the Internet as a communication tool and its pedagogic value will be underlined, preparing the younger generations to the challenges of the third millennium.
Forewords: Relations with the other VOICE Projects
The Netd@ys VOICE Project is proposed as an extension of three other VOICE Projects co-ordinated by JRC-ISIS. The aims of these Projects are to investigate into the possible applications of voice to text recognition for the deaf and to give awareness in this field to all the persons who might collaborate in testing and improving the systems.
The first Project is the JRC-ISIS Exploratory Research VOICE Project and is more concerned with the technical aspects of speech recognition for generating subtitles of conferences and telephone conversation. (Please refer to Annex 2, while the Annex 1 presents an introduction on the state of the art, the user needs and the possible applications of speech recognition for the deaf.)
The second is the Telematics Applications VOICE Project and is more aimed at organising conferences for spreading information and awareness on the use of voice to text recognition systems for the deaf. (Please refer to Annex 3.)
The Partners of the two Projects are aware of the important role of the schools in defining new pedagogic approach for the use of innovative methods of communication. They are also aware of the importance of graphical and subtitling aids in the education of young deaf students for acquiring as early as possible a good language development. Hearing impairment is a particularly important disability to be taken into consideration since it affects people of all ages. All the possible help has to be offered to a deaf student, all along his attendance to the school, for training his use of the written and oral language and for limiting his isolation in the classroom.
For these reasons the Partners addressed a considerable part of their time to the contacts with schools in order to establish collaboration for future common activities. This VOICE-School Project has been proposed by JRC-ISIS to schools. Its budget is partially covered by the JRC-ISIS Exploratory Research budget. (Please refer to Annex 4.)
Nevertheless these activities do not fit into the precise limits of the present VOICE Projects. The schools are welcome to make use of the achievements of the JRC-ISIS Exploratory Research VOICE Project and to attend the conferences that will be organised by the Telematics Applications VOICE Project. But they should participate just as any other user wishing to get more information on the present achievements, the foreseen developments and the possible applications. The budget limits do not offer the possibility of performing specific activities with the aim of helping the schools in a deeper involvement in the tests.
The motivation of the schools in participating is high, but several difficulties have to be overcome. Several specific points need further investigation and some specific help has to be provided for the schools wishing to join the Project. The pedagogic aspects have to be investigated so as to guarantee the success of this new technical approach, and in case modify some technical options of the present prototype demonstrator.
These specific activities could receive an important push if there will be the opportunity of organising them in the frame of DG-XXII actions. The Netd@ys week might be an important starting point for spreading information and co-ordinating new activities. Considering the experience already gained by the Partners, it will be rather easy to define a more precise collaboration with DG-XXII Netd@ys responsibles, in view of ensuring the maximum flexibility in following their suggestions. Roles may be defined and partners may be chosen in a large number of interested participants, considering their specific experiences in relation with the tasks that will be decided more precisely.
The Partners of the Project represent different sectors of experience and of activity (research, universities, schools, associations of the deaf and of their families, Institute for the deaf, software houses) and may ensure the complementary skills in order to cover all the aspects of the Project. The schools that are participating to the first phase of the Project represent different interests and approaches: normal schools with also deaf students or special schools for the deaf, and use of the prototype demonstrator for subtitling school lessons for the benefit of deaf students or language lessons for the benefit of any student. ALFA, AFA and IHSB, whose members are more than one thousand, represent three different ways of approaching the problems of deafness, due to different culture and language aspects. The expertise of the Partners, the previous analysis of the user needs, the availability of laboratories (hardware/software) as well as of demonstrations, the experience in organising meetings and workshops, will help in achieving the goals of the Project.
Hardware/software installation and help in the first phases
The prototype applications of voice to text recognition systems developed by the VOICE Projects are based, as far as possible, on hardware and software commonly available on the market, following the design for all principle. This allows to reduce the development costs and times as well as the future maintenance of the products. Moreover, this helps in improving the quality of products for the normal market, for any user, eliminating new barriers that often are created by new information technology tools. The cost of some voice to text recognition commercial packages is about 100 ECU (10% of the original price). The approximate cost of the basic structure for the application (a Pentium PC 200 MMX with 64Mbyte Ram, Cd-Rom and Soundblaster 16 Sound card) is 1500 ECU. This includes a fully operational PC that can be used in many other useful ways. A great advantage is also the fact that the system is not dependent on any particular company or software release.
The schools and the associations of the hearing impaired that are participating to the Project have already installed the necessary hardware and software and are starting to use it. The present configuration, even if based on standard products, requires care on several details and help by experienced persons. The training of speech recognition packages takes about one hour in order to create the user profile, so that the PC may recognise the voice of a new user. But, after having completed that phase, the user has to spend several hours to get familiar with a slight different way of speaking and of controlling the system by his voice. And probably several days are necessary in order to create a good acceptance and interaction in all the concerned persons in the classroom.
A regular help has to be foreseen during all these phases. A person with a good experience of the system has to be present, or at least available on demand in a very short time. The presence of a technician is necessary during the installation and the first tests, thus allowing the users to concentrate themselves on the pedagogic aspects of the use of the system. The presence of a person with pedagogic experience is then important for suggesting how to use the system and how to organise the lessons and the tests in the most efficient way. The Netd@ys VOICE Project will provide the users with this help, directly and via the Netda@ys VOICE Help Desk.
The transfer of information among the different schools participating to the tests is vital. The already existing VOICE Forum will make available space for creating new pages for the Netd@ys VOICE Project. The VOICE Special Interest User Group, which will be created before the ICCHP Conference in Vienna (August 98) together with its VOICE Discussion Forum, will welcome new users from the schools. A VOICE Chat Line, which has been created in April 98 and will be announced in May 98, will be available for teachers and students, for connection and Net-meetings among different schools and at home. Its use will be encouraged for activities related to the Project or related to school and also for private contacts. This will be of the utmost importance for establishing the communication of deaf students with their hearing friends for homework and private meetings. All these ways of accessing the Internet will constitute different opportunities of contacting others and sharing common experiences, ideas and suggestions.
A VOICE Help Desk will be created for the specific needs of the Netd@ys VOICE Project and will provide a constant help for the newcomers in the use of voice to text recognition systems. It will be based on a set of Web pages dedicated to explain the basic principles of the use of the Internet. A welcome page will offer to the users the choice between: information on the use of the Internet (Internet, modem, contract with a provider, how to make a search, e-mail, discussion forum, chat line); voice to text recognition (theoretical principles and practical aspects of its use); the VOICE Project (generalities, a demo, how to participate and how to get in touch with the Partners and the schools); the on-line Netd@ys VOICE Help Desk (how to contact the Help Desk and get answers on specific questions).
The Help Desk will make use of all the available Internet tools. It will also be accessible via telephone, fax and traditional mail, in order to send information on how to start using the Internet. Nevertheless, attention will be paid for driving the users to start as soon as possible the use of the Help Desk via the Internet. The access will be guided towards the Help Desk's correspondents with the more appropriate technical, pedagogical and linguistic competence.
Working together on the Internet
The difficulties encountered in a class with a deaf student may get some answer in far locations. Someone who has already worked on that problem may be now working on some other theme or in some other town. And a teacher or a parent who has to invest his time on a specific case, could be more motivated to do so, knowing that his work could be of help for similar cases in other schools, at present or in the future.
Several schools as well as several H-Groups of the provincial super-intendents of public education, even if with great experience in disability, have not been able to participate so far to activities related to the use of the Internet. The creation of a new network, where these particular needs are felt at functional level, will help in spreading the awareness of the pedagogical value of the use of Internet in schools in fields related to disability.
An example of this is the following case. Often the teachers organise part of their lessons on the use of modern means of communication, as software packages or videotapes. Although not completely useless for the deaf, these means may generate frustration in that only a part of the information they provide is in a form accessible to a deaf student. For this reason, they may increase his isolation in the classroom. Some associations of the deaf, as for instance the Centro dei Servizi Silenziosi of the Ente Nazionale Sordi in Milan, has more than 3000 subtitled video-cassettes (films, documentaries, pedagogic programmes) and may offer them to the schools. The list of these video-cassettes is accessible through the VOICE Forum since the beginning of 1997, but just a few schools are informed of this service and know how to make the best use of it. This lack of information exists in many other similar cases.
More regular and intensive use of the Internet
The use of the Internet is proposed here not only in relation with the foreseen tests of voice to text applications. There is, behind that, a broader idea of a more regular and intensive use of the Internet at schools. The Netd@ys VOICE Project might be a particularly motivating reason for starting new applications and to spread the use of the Internet. Deaf students, who will communicate among them in different schools or at home and retrieve information from the Internet, will be an example for other students. The teachers starting to use the Internet for communicating among them in different schools and with some specialised centre of reference (deafness, multimedia, etc.), will encourage their colleagues to access the Internet more frequently.
Some schools, as for instance ITSOS in Milan, have a good experience in the use of the Internet and have their own Web site. But several schools are not yet at this stage and they have to be encouraged even to buy a PC or a modem and to sign a contract for the connection to the Internet. Others, even if connected to the Internet, need some information on the practical use of the system and on how it is possible to create and hold regular contacts with other schools through the Internet. Some schools are aware of the ways of creating and managing a Web site, but are looking for more information and for some successful example to follow.
Sometimes, even when a school has a connection to the Internet, there are several practical difficulties to overcome before accessing the Internet (PC and classroom availability, timetable, telephone line, passwords, etc.). And when, at the end, a student or a teacher is on the Internet, he or she may feel uncertain about the procedure to follow and may not find a precise reference point, so interesting as to be motivated to reconnect a second time later on. In that case, the fact that the school is involved in a project related to the Internet may be particularly motivating. The Web page of the Project, the e-mail addresses of some correspondents and all the other tools here proposed, may constitute a valid starting point to continue the connection and find new paths to follow.
The VOICE Forum and the Netd@ys VOICE Help Desk (and if applicable the other proposed VOICE Internet tools) will be presented with choices in several languages (English, Italian, German, at starting, possibly French). And, even more important alternative, also the text description of images and tables will be offered. This approach will follow the Trace Centre's rules for the accessibility to the Internet for the blind (via text to voice synthesis) and for people with reduced mobility of their hands in using the mouse.
The new generations of students are accepting the idea of working in Europe together, using several languages. This idea will be extended so far that, exactly at the same level on the screen where the choice will be offered between flags of several languages, also additional identical flags will be available for users with special needs, as for instance a text to voice synthesis for the blind.
Even if the first objective of the Project is to provide a help to the deaf, a second aim is that of awareness-raising of the importance of any disability. The general idea that should be transmitted by the Netd@ys VOICE Project is that, at a European scale, the problems of people with disabilities in the school and in the society are considered of the utmost importance. In this vision, the use of the Internet may provide a particular help for uniting people of different areas working together with a common objective.
This aspect is particularly important for encouraging the deaf to communicate with their hearing friends. Actually, the name VOICE is used for the Project in order to spread the idea that the Internet is also the VOICE, the written voice of a teacher and the written voice of friends trough the Internet, for those who can not hear the spoken voice. And the idea is even larger, in the sense that also the blind may read the spoken Web pages via text to voice synthesis of the written Web pages.
Visibility in the Netd@ys Week
In order to present to a larger public the information on the availability of above-mentioned systems, during the Netd@ys week several initiatives will be carried out.
Several presentations of the VOICE Project and the possible applications of voice to text recognition will be organised. The prototype will be used for demonstrating the subtitling of school lessons for the benefit of deaf students and of language lessons for the benefit of any student. The presentations will be held at the schools that are already using the prototype and will be organised as an open doors week addressed to visiting teachers and students from other schools.
A videoconference will connect among them the schools that are already using the prototype. This connection will be open to other schools looking for information on the system or wishing to learn how to use it. The videoconference will be held in the appropriate time, so that a television broadcaster will transmit a part of it during a program for the schools. Should they wish to do so, the officials of DG-XXII in charge of Netd@ys will participate via videoconference to the television transmission, from the EC offices in Brussels.
A pamphlet on school and deafness will be prepared and distributed during the presentations to people attending them directly, while those attending trough videoconference or television transmissions will be informed of the Internet address of the VOICE Forum, where the same documentation will be available in an appropriate format.
The ITSOS school in Milan will organise a Workshop on voice recognition and multimedia applications, intended as training course for teachers, with the agreement and the collaboration of the provincial super-intendent of public education. The aim of announcing the Workshop is, of course, that people come and attend it. But, as a second aim, the information that will be provided through all the provincial super-intendents of public education will at least guarantee that even those who will not attend will be informed about the Netd@ys Project and about the address of the Project's Web site.
The Netd@ys VOICE Help Desk, created specifically in view of this week, will be announced. Its use will be encouraged for receiving on line help on the use of applications of voice to text recognition and on the use of the Internet.
The above-mentioned activities will be organised mostly in Italy and Austria, trying to enlarge the field by the videoconference, the television transmission and the contacts with schools and associations in other countries. A videotape and/or Cd-Rom will be created at that moment and will be available for further reference for the new comers.
At a larger level we will stimulate and co-ordinate a world Internet search.
The search will have the objective of finding other similar applications in the world and of getting in touch with schools performing similar tests using voice to text recognition systems or with associations of the deaf.
This will help in: spreading the access to the VOICE Forum for providing information on possible applications of voice to text recognition in the schools; facilitating the starting of new contacts via the VOICE Special Interest User Group; encouraging the discussion of the technical and pedagogic aspects of voice to text recognition via the VOICE Discussion Forum; accessing the Netd@ys VOICE Help Desk; stimulating the use of the VOICE Chat Line for exchanging chatting and informal messages on aspects related to the Project and also in other fields. This aspect is particularly important for encouraging the deaf to communicate with their hearing friends. And this should work also in the other sense round, that is that (at least once !) the hearing students will have to ask for the collaboration of their deaf friends in order to get more information for the search.
The impact that such a Project may have is enormous, in underling the problems of disabled people at school and in several aspects of every day life. This awareness-raising process will not be based just on a passive pietism, but will start from the most interesting and modern approach, that of the use of the latest new technology applications, like speech recognition. So far, independently of the success of the proposed technical solutions, the Netd@ys VOICE Project will have an immediate important effect. The final goal is broader than the technical aspects related to speech recognition. The aim is that of a large awareness campaign, which will increase the awareness of teachers, students, families, people at home, on the problems of the disabled and of their integration in the school and in the society, in the everyday life.
The Project will improve the accessibility to information, offering an additional means to participate fully in the information society and improving the quality of life of persons who at present have difficult access to information and communication. A wider diffusion of subtitles and of the Internet tools will greatly increase the interaction of the deaf with each other as well as with the society in which they live.
An easier access to schools and universities and more contacts through the Internet will allow a more satisfying life and also a better choice of a work corresponding to personal capabilities and, at large, more economic productivity for the society. When more conferences, meetings and discussions slowly become subtitled, there will be an increasing in participation from the hearing impaired community. This contribution will increase the effect that their decisions have on the surrounding environment, which will subsequently improve their standard of living. Once started this improvement of their integration and interaction in the society will have a snow ball effect and it is therefore: this initial push that is so vital. The use of the Internet as a communication tool and its pedagogic value will be underlined, preparing the younger generations to the challenges of the third millennium.