An Assistive Mobile System Supporting Blind and Visual Impaired People when Are Outdoor T. Valls Mataró, F. Masulli, S. Rovetta, A. Cabri, C. Traverso, E. Capris, S. Torretta Abstract— In this paper we present the TARSIUS system, based on mobile technology and aimed at enhancing visuallyimpaired and blind people’s capabilities in visual scene understanding and geolocation while are outdoor. The system components are the TARSIUS app for mobile devices, a web server, and the Remote Assistance Center. Its interface is optimized for the perceptual characteristics of its users. Moreover, the TARSIUS navigation sub-system not only leverages the GPS system, but also Bluetooth LE/iBeacon tags placed along the streets at points of interest and dangerous paths and areas. I. I NTRODUCTION CCORDING to the World Health Organization , , 285 million people are estimated to be visually impaired worldwide: 39 million are blind and 246 have low vision. However, refractive error as a cause of visual impairment was not included, which implies that the actual magnitude of visual impairment is higher. More than 82% of blind people are 50 years or older, although they represent only 19% of the world’s population. With an increasing elderly population in many countries, even more people will be at risk of visual impairment due to chronic eye diseases and ageing processes. Visual impairment is not distributed uniformly throughout the world. More than 90% of the world’s visually impaired live in low-income settings and mostly in developing countries. Moreover, females have a significantly higher risk of being visually impaired than males. Worldwide, 19 millions children are estimated to be visually impaired; of these, 12 millions children are visually impaired due to refractive errors, a condition that could be easily diagnosed and corrected. Eventually 1.4 million are irreversibly blind for the rest of their lives and need visual rehabilitation interventions for a full psychological and personal development. The principal causes of visual impairment are: (a) uncorrected refractive errors: myopia, hyperopia or astigmatism (43 %), (b) unoperated cataract (33%), and (c) glaucoma (2%). Globally, 80% of all visual impairment can be prevented or cured. Nearly one-third of people with vision loss suffer from clinical depression, a rate that is twice as high as the general A Authors affiliations: Toni Valls Mataró(1), Francesco Masulli(2), Stefano Rovetta(2), Alberto Cabri(2), Carlo Traverso(3), Elisabetta Capris(4), Simone Torretta(4). (1) Cultural Association trescucarachas, Barcelona, Spain (email: [email protected]); (2) Dipartimento di Informatica, Bioingegneria, Robotica e Ingegneria dei Sistemi (DIBRIS) Università di Genova, Genoa, Italy (email: [email protected], [email protected], [email protected]); (3) Dipartimento di Neuroscienze, Riabilitazione, Oftalmologia, Genetica e Scienze MaternoInfantili (DINOGMI) Università di Genova, Genoa, Italy (email: [email protected]); (4) David Chiossone Onlus per Ciechi e Ipovedenti, Genoa, Italy (email: [email protected], [email protected]). 978-1-5386-3906-1/17/$31.00 ©2017 IEEE population of the same age . Moreover, injuries from falls are common with visually impaired patients, and they tend to stay in the hospital nearly 2.5 days longer than patients with normal vision . Overall, visual impairment worldwide has decreased since 1990s, despite an ageing global elderly population. This decrease is mainly the result of a reduction in visual impairment from infectious diseases through: (a) overall socioeconomic development; (b) concerted public health action; (c) increased availability of eye care services; (d) awareness of the general population about solutions to the problems related to visual impairment (surgery, refraction devices, etc.). Nevertheless, the worldwide direct and indirect costs of visual impairment are growing, as shown in Fig. 1. The population of people with visual disabilities is very heterogeneous, with different needs and degrees of reduction of autonomy. For example: • Less than 20% of people with visual impairments understand the Braille code ; • Many visually impaired people don’t like to use the white cane or the guide dog for social inclusion reasons; • In situations of confusion, due, e.g., to construction of a new road or to the change of location of a bus stop, the visually impaired person needs actual help; • In severe cases of peripheral vision loss, but with central residual vision, the use of a wide angle lens can increase the field of view up to a 30%; • Portable digital magnifiers allow to freeze the complete picture of the scene on the screen that later can be analyzed by the low vision user using personalized color filters to enhance the contrast; 1 • 52% of the visually impaired people with age between 1 Data concerning the members of the Organizaciòn Nacional de Ciegos Españoles Fig. 1. Worldwide costs of visual impairment (from ). Fig. 2. Some apps/systems for blind and visually impaired people 17 and 64 are using or are expected to use in the near future smartphones or tablets. About 80% of them have a residual visual functionality sufficient for moving away from home, for housework and for reading. These people are interested in using mobile devices in daily outdoor life both as telephone (essential for receiving help in case of difficulty) and for further support through an app2 , though at present there are still no effectively useful apps. At present few apps for blind and visual impaired people are available, and usually are aimed at supporting one mobile device function only. Some of them are part of the operating systems accessibility services, including: (a) screen readers that let the user know by vocal messages what is happening on the touch-screen; (b) readers for emails, web pages, and books; (c) intelligent assistants, working as vocal calendar, placing phone calls, configuring other apps; (d) dictaphones converting user’s words into text; (e) apps for font adjustment, screen zooming, and screen color management. Additional apps, to be installed by the user, provide assistive tools to: (a) support incoming and outgoing phone calls; (b) provide aid to navigation in town streets; (c) turn the phone in a digital magnifying glass equipped with a flashlight; (d) read plates and billboards by OCR; (e) quickly call taxis or emergency services. Among the available apps/systems for blind and visually impaired people, some of the more interesting are (see Fig. 2): • Loowi suite (Android) that gives access to the basic functions of the smartphone in an intuitive and easy way 2 A (mobile) app is a software designed to run on mobile handsets such as smartphones and tablets. • • • • with enhanced graphics, voice information and vibration system, and among its features and functionalities includes the possibility to obtain the user position through GPS (geolocation). Ariadne GPS (iOS) that includes talking maps allowing the user to explore the surrounding world by moving her/his finger around the map, haptic feedback while the users is crossing a street, and bus or train stops announcements. GeorgiePhone app family (Android) that, in additions to the features included in Loowi suite and Ariadne GPS and others, include a GPS navigator using step by step spoken or on-screen interaction, an app for finding points of interest close to the user, a helper activating the remote assistance that sends a text message to carers and friends connected via an ad-hoc app, an app for bus stops reaching, bus approaching selection, and bus get off, and an app for quick taxi calling. GoAll app (iOS and Android) that allows deaf-blind people to receive and enjoy television content without intermediaries. It is based on the PervasiveSUBsoftwarethat compiles all the subtitles of television channels and sends them to a central server. The GoAll app retrives the subtitles from the central server and can sent tehm to the braille line of the deaf-blind person. Sesamonet (Window Mobile) app that includes a database with information on the location and makes use of RFID (Radio Frequency Identification) passive transponders (no requirement of power supply) technology to create a path guiding visually impaired persons through a location. The walking stick includes an embedded antenna (with a bluetooth transmitter) which detects/reads the RFID transponders and the app smart- Normal vision Blurred vision Loss of peripheral vision Loss of central vision Fig. 3. • • Normal and pathological vision (simulation). phone a database. XSIGHT Smart glasses that detect nearby obstacles and present them in a simple high contrast way, including an infra-red projector, an Infra-red camera a depth camera, and Organic Light Emitting Diodes (OLED) lens. Eyra Horus wearable assistant that recognizes faces, objects, obstacles and describes the world. It is composed by two cameras, a bone conduction headset and GPUaccelerated computer vision using deep learning. Some features here described are useful for supporting blind and visually impaired people walking alone in the streets, but it is necessary to better integrate them to improve their usability. Moreover, the interfaces should be optimized for the specific perceptual characteristics of the users and the navigation system should be more resilient and precise, as at present it works only when a good GPS connection is present in addition to Internet connectivity. In this paper we present the TARSIUS3 project aimed at the development of an ICT assistive infrastructure for visually impaired and blind people support based on mobile technology (smartphone e tablet) to enhance their ability to understand a visual scene and to orientate while are outdoor. The project, that is in the initial phase of development, involves at present the University of Genoa (https://www.unige.it/), the Istituto David Chiossone (http://www.chiossone.it/), the Unione Italiana dei Ciechi e degli Ipovedenti (https://www.uiciechi.it/), and the Associazione per la Retinite Pigmentosa RP Liguria (http://www.rpliguria.it/). Additional partners are welcome to join to the working group. In the remainder of this paper, Section II presents the apps currently available for the blind and visually impaired people; Section II discuss some perceptual constraints to be considered for software design; then in Section IV analyzes the possible low-cost radio-tagging technologies that can be 3 The Tarsius is a small Asian primate that has evolved strengthening its night and twilight vision. exploited in the geolocation and navigation functions of our system; Section V presents the TARSIUS system; discussion and conclusions are in Section VI. II. P ERCEPTION IN B LIND AND V ISUALLY I MPAIRED The spectrum of visual impairment includes a number of conditions with diverse characteristics. In general it is necessary to distinguish among the following situations: • • • Complete blindness: the person does not perceive any visual stimuli; Legal blindness: there is a residual vision below a conventionally established minimum; Low vision: there is a deficient visual capacity in one or more aspects, for example in the extension the visual field. Fig. 3 shows a simulation of the visual field of people with normal vision, compared to some visual pathologies, like blurred vision, loss of peripheral vision and loss of central vision. In people with low vision, the optimal visual contrast can be obtained with a combination of colors that varies individual from individual . Fig. 4 shows the output of some filters that may give the optimal color contrast for a particular individual. An app for blind and visually impaired people support should included an initialization phase for detecting the optimal color contrasts and the filter that will be used in the supportive functions. Another issue concerning the user’s perceptual system is the usage of voice systems for the user interface. It is important to avoid using standard headphones or earphones that isolate the user from the environmental sounds and voices which are the main informative channel for blind or visually impaired people. Bone conduction headphones permit to receive both audio channels, but force the user to carry an additional item. Fig. 4. Examples of personalized filters for enhancing vision. Differentiated mobile vibrations, coding haptic information, can be fruitfully exploited for many simple man/mobilehandset communication tasks. III. O UTDOOR G EOLOCALIZATION The Global Positioning System (GPS) is a space-based navigation system used for car navigators that provides location and time information in all weather conditions, anywhere on or near the Earth where there is an unobstructed line of sight to four or more GPS satellites. The precision of the GPS system is limited to an interval of one or few meters . A navigation system supporting blind and visually impaired individuals when moving outdoor in a town must therefore integrate some ad-hoc short-range navigation system. One solution is radio tags positioned close or on the points of interest (e.g., shops) and dangers (e.g., roadworks) by the local government or by public and private organizations and individual citizens. The possible low-cost short-range radio-tagging technologies that can be massively distributed to this aim in the urban scene are: • Radio-frequency identification (RFID) transponders, wireless access tags attached to objects . • Near field communication (NFC), radio electronic devices that establish communication when they are to a distance of 10 cm or less , based on the RFID technology. • Bluetooth low energy (Bluetooth LE) that is a wireless personal area network technology intended to provide, at a considerably reduced power consumption and cost, a communication range up a maximum of 100 m . • iBeacon (beacon) that is a class of Bluetooth LE devices that broadcast their identifier to nearby portable electronic devices and are optimized for localization tasks. They usually have the size of a 2 Euro coin and their button battery, their largest-sized part, can last up to two year . Generic RFID transponders are not integrated in most mobile handsets, while NFC and Bluetooth LE are embedded in most recent smartphones. Given their communication range and low energy consumption, Bluetooth LE and iBeacon are the most promising radio-tagging technologies to be integrated with GPS in navigator systems for visual impaired people. IV. T HE TARSIUS S YSTEM The goal of the work described here is to design and deploy an integrated system that includes a complete selection of the functionalities described in the previous paragraphs, both for exploration and for navigation, giving access to all of them through a common interface. The TARSIUS system is based on the following components: (A) TARSIUS app for mobile devices, providing the main interaction with the system; (B) Web Server; (C) Remote Assistance Center (social network). These components will be described in the following subsections. A. TARSIUS App In the design of the TARSIUS System particular care is dedicated to the design of a ”user friendly” interface that takes into account the peculiarities of the users. The functional buttons are placed at the borders of the touchscreen, with the most important ones placed close to the corners, and the SETUP button centered on the screen. The man-device interaction occurs through tactile feedback, sounds and speech for all visually impaired users. For each button a differentiated feedback is provided, that can consists in audio (sound and voice) or as vibration patterns. Audio and vibration feedbacks can be enabled or disabled independently. It is important to highlight that even the haptic feedback gives a different vibration response depending on the feature associated to the button on the touch-screen. (a) A pharmacy and a post-box (b) A bus coming (c) Waste containers: paper, glass, plastic (d) Roadwork Fig. 5. Outdoor scenarios and possible tags. (simulation). The TARSIUS app includes the following main functions activated by pressing the relevant button on the touch-screen: • • • SETUP procedure that occurs at the first run of the TARSIUS app, or on pressing the SETUP button. It allow the user to enter his personal data, to configure the audio and tactile feedbacks, to pre-set the options of the different selectable functions. In the case of lowvision users, the SETUP allows them to customize the colors and features of the touchscreen to obtain the optimal visual contrast and to select the optimal contrast filter for enhancing the perception of the camera frozen pictures (see Sect. II). NAVIGATION System (NS) that integrates GPS, NFC, RFID, Bluetooth BLE, and iBeacon technologies for user geolocation and identification of possible points of interest (e.g., shops) and dangers (roadworks). The user selects the destination and the system interacts with the user through coded vibrations, sounds and simple vocal messages (e.g., ”forward”, ”turn to the right”, etc.) guiding the user to reach the goal. In case of a danger or an obstacle, a special warning tone notifies the user. In addition, the NS is able to warn the user that a bus is coming to a stop, to notify the user the different stops during the bus riding, to warn him about the presence of a staircase and the number of its steps and support him in many tasks that are necessary for moving outdoors. It allows the user to introduce new personal points of interest such as the house of his family, of his friends, a gym, a pharmacy. HELP System (HS) that is started by pressing a button, • • allowing the user to call the Remote Service Center in case of danger or confusion, when the user does not understand what is around (e.g., when a path has been modified by roadworks). After pressing this key, the user may get in contact with a trusted professional or an operator, who, via the ”remote control”, can help the user understanding what’s around and solve the problem with the aid of the operator’s instructions. The operator can understand the situation thanks to the activation of the phone camera, seeing what is around the person and providing help in locating obstacles, the name of the street, and so on. EMERGENCY System (ES) that is started by pressing a button for a certain number of seconds, for obtaining immediate help in case of an accident, fire, and so on. VISION System (VS) constituted by a digital video magnifier which allows the user to freeze the surrounding scene as an image on the screen for a later analysis, exploiting at best the residual vision capabilities making use also of personalized color filters enhancing the contrast and possibly zooming the visual scene. The picture can be also annotated with few seconds of audio recording that allows the user to remember the photo, the environment and her/his feelings. On user’s request the image is tagged with information on attractions and dangers (augmented reality); it is also possible to process selected regions of the visual scene with algorithms for Optical Character Recognition (OCR) and, depending on settings, the voice system will pronounce those information on demand. In Fig. 5 some typical • sceneries are presented. The view in Fig. 5(a) is tagged with the indications of ”pharmacy” and ”post box”. The tag in the view of Fig. 5(b) indicates the bus and is complemented by its number and direction; to achieve this the TARSIUS system may interact with the bus data base of the local government. The positioning of some iBeacons allows the TARSIUS system to label the three waste containers correctly in the scene of Fig. 5(c): ”paper”, ”glass”, ”plastic”. An object recognition algorithm can highlight with a suitable color the handle for opening the container. An iBeacon allows the TARSIUS system to warn the user about the danger in Fig. 5(d). NEWS System (NS) that updates the user about news and useful information. B. WEB Server The WEB Server contains an updated map of the town including the available tags and signs for geolocation (sites of interest, obstacles and dangers). Its main tasks are: • Geolocating the users of the TARSIUS App and recoding a log of their activities; • Supporting the TARSIUS App with contents and remote computation. C. Remote Assistance Center The Remote Assistance Center (RAC) is supported by the local government and/or national health service and/or private and/or public organization and family members and caregivers included by the final users in their personal social networks during the SETUP procedure and its composition can be changed. The RAC has the following functions: • Geolocates the users of the TARSIUS app and collects the logs of their activities; • Receives requests for support / help from the users; • Provides remote advice and remote assistance through the camera of the mobile device and any available cameras in the area; • Requests the public administration to install new Bluetooth or iBeacon tags for geolocation. V. D ISCUSSION AND C ONCLUSIONS Due to its nature of integrated infrastructure, the TARSIUS system represents a valuable asset for social inclusion of blind and visually impaired people who will benefit from the increasing autonomy that the system is fostering. Even if the project is in an initial phase, we believe that its potential is high and it will get even higher as new technologies will become available in the near future. That’s why new partners are welcome to join our team. R EFERENCES  AMD Alliance International, “The global economic cost of visual impairment.”, 2010. Internet document http://www.icoph.org/resources/146/The-Global-Economic-Costof-Visual-Impairment.html Last access Aug 22, 2015.  L. Braille, “Nouveau procede pour representer des points la forme meme des letters, les cartes de geographie, les figures de geometrie, les caracteres de musiques, etc., a l’usage des aveugles,” Institution royale des jeunes aveugles, 1839.  V. Coskun, K. Ok, B. Ozdenizci, Near Field Communication (NFC): From Theory to Practice. Wiley, 2012.  M. S. Gast, Building Applications with iBeacon: Proximity and Location Services with Bluetooth Low Energy. O’Reilly Media, Inc., 2014.  B. Glover, RFID Essentials. O’Reilly Media, Inc., 2006.  J.E. Javitt, Z. Zhou, R.J. Willke, “Association between vision loss and higher medical care costs in Medicare beneficiaries costs are greater for those with progressive vision loss,” Ophthalmology, 114(2), 238245, 2007.  B. Jenny, N. Vaughn Kelso, ”Color Design for the Color Vision Impaired,” Cartographic Perspectives 61/58, 2007.  A.R. Morse, E. Yatzkan, B. Berberich, R.R. Arons, ” Acute care hospital utilization by patients with visual impairment, Archives of Ophthalmology, 117, 943-949, 1999.  K. Townsend, C. Cufı́, Akiba, R. Davidson, Getting Started with Bluetooth Low Energy. O’Reilly Media, Inc., 2014.  World Health Organization, ”Magnitude of blindness and visual impairment.” Internet document http://www.who.int/blindness/causes/magnitude/en/, Last access Aug 22, 2015.  World Health Organization, “Visual impairment and blindness.” Internet document http://www.who.int/mediacentre/factsheets/fs282/en/, Last access Aug 22, 2015.  G. Xu, GPS: Theory, Algorithms and Applications. Springer-Verlag, 2007.