AI-based Railway Obstacle Intrusion Detection System with Multimodal Transformers

Principle Investigator:
Ir Prof. NI Yiqing, Yim, Mak, Kwok & Chung Professor in Smart Structures; Chair Professor of Smart Structures and Rail Transit; Director of National Rail Transit Electrification and Automation Engineering Technology Research Centre (Hong Kong Branch); Director of The Hong Kong Polytechnic University-Hangzhou Technology and Innovation Research Institute

Rail accidents caused by obstacles on the tracks are a major concern. Developing intelligent obstacle intrusion detection systems (OIDS) is crucial for train safety. Our advanced OIDS has three main components: (1) sensors, including a camera and LiDAR, (2) a real-time data collection and warning module, and (3) a transformer-based detection model.

First, the visual sensors are calibrated for optimal performance and mounted on locomotives. The collected multimodal data are then synchronised and fed into the transformer-based detection model. This model extracts features from both images and point clouds, detecting obstacles that are currently or potentially encroaching on the rail area by analysing the combined data. Based on the detection results, real-time warnings are sent to operating trains to prevent potential accidents.

The transformer model is trained using both real and synthetic samples in different weather and lighting conditions, enhancing the robustness and versatility of the OIDS in diverse scenarios.