PAIR Distinguished Lecture: Prof. ZHANG Jie of University of Science and Technology of China delivers “Machine Listening: Extending AI from Vision to Sound and Vibration Sensing”

Abstract

While advances in artificial intelligence have largely centered on vision-based technologies, the complementary domain of machine listening — the machine interpretation of sound and vibration signals — remains underexplored. Just as human perception relies on both vision and hearing, integrating these modalities can provide a more complete understanding of complex environments. The development of machine listening directly impacts robotics, autonomous driving, and a broad range of industries. Broadband seismic sensors extend auditory capabilities far beyond human perception, capturing both acoustic pressure waves in air or fluids and elastic waves propagating through solids across a wide frequency spectrum. These sensors can detect vibrations as subtle as those produced by a pencil breaking, enabling real-time monitoring through the integration of physics-based models, deep learning, and large-scale data analysis. We introduce a joint visual–seismic sensing framework that generates detailed digital records of transportation systems, infrastructure health, human activities, and environmental processes. This “superhuman hearing” capability opens pathways to transformative applications. By bridging vision and listening, machine listening advances the frontier of AI-enabled sensing, fostering societal resilience and sustainability.