PolyU develops smart assistant “AI-Care” that automatically identifies daily tasks and provides guidance to support independent living for visually impaired people
Hong Kong currently has nearly 200,000 visually impaired people. Although white canes and guide dogs can assist with basic mobility, daily tasks such as shopping and taking public transport remain challenging for them. A research team at The Hong Kong Polytechnic University (PolyU) has developed an innovative assistive system, “AI-Care”,which is driven by an advanced artificial intelligence (AI) model. Connecting smart glasses and a mobile application, the system proactively detects surrounding objects and provides voice guidanceto help visually impaired people handle daily tasks independently. The team has partnered with industry collaborator Hillming Smartech Ltd. (HSL) to advance commercialisation, and with the Hong Kong Society for the Blind (HKSB) to launch community trials, aiming to inject new momentum into building an inclusive smart city.
The launch ceremony of the AI-Care project was held yesterday (10 June) at the PolyU campus. It was officiated by Prof. the Hon. Kam-fai WONG, Legislative Council Member of the Hong Kong Special Administrative Region (HKSAR) of the People’s Republic of China; Prof. Changwen CHEN, Interim Dean of the PolyU Faculty of Computer and Mathematical Sciences; Prof. Qing LI, Head of the PolyU Department of Computing; alongside Mr Sean FONG, Chief Executive of HKSB; Mr YANG Zhiming, Technical Director of Assistive Technology of the China Culture and Information Service Center of the Visually Handicapped (China Braille Library); Mr Hillming LI, CEO of HSL; and Ms Shirley TSANG, Director of Rehabilitation Services at HKSB. Project leader Prof. Jing LI, Associate Professor of the PolyU Department of Computing and Associate Director of the PolyU Research Institute for Generative AI also attended the event.
Addressing the ceremony, Prof. Changwen Chen said, “PolyU is committed to impactful research. We believe the true measure of technological innovation lies not only in how smart our AI becomes, but also in how deeply it empowers those who need it most. By transforming AI from a passive tool into a proactive companion, AI-Care not just advances the development of assistive technology, but also promotes social inclusion.”
Mr Sean Fong remarked, “AI-Care empowers people with visual impairments to transcend physical limitations, enabling them to live more independently and participate in society with confidence. Many of HKSB’s service users have already benefited from such innovations, and we are eager to integrate this novel system into our services.”
Mr Hillming Li said, “The Government has been a strong advocate for building an accessible city. AI-Care is not only a successful transfer of cutting-edge technology, but also an innovation with a human touch, empowering underserved communities to experience the world more fully and embrace greater opportunities for a richer and more fulfilling life. We hope to further expand its application and even scale this technology across the country.”
AI-Care is built upon an advanced AI model that is adapted from cutting-edge Visual Question Answering and Natural Language Processing frameworks specifically for daily shopping scenarios, covering the full workflow from product searching and payment to receipt audit. Breaking away from the passive “question-and-answer” approach of traditional digital assistive tools, it uses smart glasses to capture the user’s perspective, thereby proactively identifying everyday objects. The system then transfers image data to the mobile application for processing and analysis, and generates real-time audio instructions.
Smart glasses available on the market mostly target sighted users, relying on visual context and sight-dependent cues, and therefore do not fully meet the unique needs of visually impaired users. AI-Care, on the other hand, adopted rigorous post-training for its AI model using domain-specific data, making its image recognition and voice guidance better aligned with the spatial perception and cognitive needs of visually impaired users, and ensuring precise, actionable guidance. The system completed phased testing in the first quarter of this year, involving 17 visually impaired participants, and achieved an average satisfaction score of 93. The team plans to scale up the trials through continued collaboration with the social welfare sector.
The research team demonstrated the current functions of AI-Care at the event. Prof. Jing Li added, “For our team, what we most look forward to is seeing how AI research can truly support independent living for the visually impaired. While current AI models show impressive generic skills, enabling them to understand users’ needs as flexibly and accurately as human assistants, to know how to provide help and to intervene proactively at the right moment remains a major challenge. AI-Care embodies a human-centric approach to technology and marks a major step forward in the development of AI-powered assistive tools.”
Supported by HK$5 million in funding from the HKSAR Government’s Innovation and Technology Fund and other industry sponsors, the research team has completed the design of the core AI model and verified the stability of the cloud-based server. The team is currently working to enhance the system’s memory management, optimise the hardware and advance prototyping, with the aim of commercialising the technology and expanding its application.
