Deep Multimodal Network Jointly Forecasts Location, Journey Time and Transport Mode

Study conducted by Prof. Hans Shuaian WANG , Prof. Min XU and their research team

The spread of smartphones, GPS devices and connected vehicles has given cities access to vast quantities of mobility data. When analysed effectively, these data can help transport authorities anticipate movement, manage congestion and provide more responsive services. A key capability is trajectory prediction: forecasting how an individual or vehicle is likely to travel based on previous movements.

Yet urban journeys are complex. As illustrated by the spatial distribution of walking, cycling, driving and public transport journeys in the Geolife dataset (Figure 1), people move at different speeds, at different times and through different parts of a city. These characteristics are interrelated, not independent.

Figure 1. Spatial distribution of trajectories by travel mode in the Geolife dataset

Most existing prediction models focus on a single task, such as forecasting a traveller’s next location, estimating journey time or identifying the transport mode used. Even multi-task models commonly predict only location and time, overlooking other useful information or combining data in relatively simple ways.

To address this limitation, Prof. Hans Shuaian WANG, Professor of the Department of Logistics and Maritime Studies at The Hong Kong Polytechnic University, together with Prof. Min XU, Associate Professor of the Department of Industrial and Systems Engineering at the same university, developed a Deep Multimodal Network (DMN) that jointly predicts future locations, travel times and transport modes by explicitly modelling relationships both within and between different types of trajectory data. The research has been published in Information Fusion [1].

Location, time and transport mode influence one another

A mobility trajectory contains several types of information. Spatial coordinates show where a traveller moves, temporal features indicate when and for how long the journey occurs, and mode data distinguish between options such as walking, cycling, driving and public transport. Each modality forms its own sequence over time, but it also interacts with the others.

Consider a person making three trips using different modes (Figure 2). Within the spatial data, previous locations help indicate likely future locations. Within the temporal data, past journey durations can provide clues about future travel times. These are intra-modal dependencies: relationships contained within the same type of information.

Figure 2. A trajectory comprising three trips with different travel modes

The modalities are also interdependent. A longer journey time will generally correspond to a greater distance, although factors such as congestion can alter that relationship. Similarly, a traveller using a car or train can usually cover a greater distance in a given period than someone walking or cycling. Knowing the time and likely mode can therefore improve location prediction, just as spatial movement can help identify the mode being used.

Earlier multi-task models have often processed each modality separately before merging the resulting features through concatenation or summation. Although such models may learn some relationships indirectly, this “brute-force” fusion does not explicitly calculate how one modality should inform another.

The research team therefore proposed an architecture that computes both intra-modal and inter-modal attention. This enables the model to identify not only patterns within location or time sequences, but also the relationships connecting them, improving all three prediction tasks within a unified system.

The network learns how different travel signals connect

The DMN is based on the Transformer model [2], which uses an attention mechanism to determine which parts of a data sequence are most relevant to a prediction. Its design consists of two main components: a feature extraction module and a multi-task prediction module. The framework is illustrated in Figure 3.

Figure 3. Framework of the Deep Multimodal Network

The feature extraction module first converts heterogeneous trajectory data into a shared numerical format. Continuous inputs, such as GPS coordinates and time of day, and discrete inputs, such as the day of the week, have different structures. Embedding layers project these inputs into a common feature space, while positional encoding preserves their order within each journey.

The embedded data then pass through two types of encoders. Local encoders learn intra-modal relationships—for example, how earlier locations relate to later locations or how previous time points influence subsequent ones. Translational encoders, the model’s central innovation, calculate inter-modal attention. They allow the network to examine spatial information in relation to temporal information, and vice versa, rather than treating the two as isolated streams.

The second component, the multi-task prediction module, uses multimodal decoders to connect historical travel patterns with possible future sequences. It generates three outputs concurrently: predicted locations, travel times and transport modes. The decoders consider relationships within each future sequence as well as connections across the different outputs.

This explicit inter-modal attention distinguishes the DMN from earlier approaches that simply concatenate or add separately extracted features. Instead of leaving a downstream predictor to discover cross-modal relationships indirectly, the network is designed to focus dynamically on the relevant connections between space, time and mode.

The model was trained end-to-end using the open-source Geolife dataset, collected by Microsoft Research Asia. It contains GPS trajectories from 182 users recorded between 2007 and 2012, covering more than 1.2 million kilometres and 48,000 hours of movement. The journeys include walking, cycling, driving and public transport.

Training used a weighted combination of regression losses for location and travel-time forecasts and a classification loss for transport-mode prediction. This allows the entire network to learn the three tasks together.

A unified model also offers a computational advantage. Separate models must each learn their own representations and complete their own training process. By sharing feature extraction and attention calculations, the DMN can draw on synergies between tasks while reducing duplicated computation. In the ablation study, training three separate task-specific versions took a combined 16,200 seconds, compared with 6,600 seconds for the full multi-task model.

Prediction gains grow on longer trajectories

On the Geolife test set, the DMN outperformed two established sequence-modelling baselines, Long Short-Term Memory (LSTM) [3] and Sequence-to-Sequence (Seq2Seq) [4], across all three tasks and all tested trajectory lengths.

Its advantage was particularly evident for longer sequences. At a trajectory length of 150, the DMN reduced location prediction error by 67% and travel-time error by 69% compared with LSTM. Against Seq2Seq, the corresponding reductions were 61% and 67%. For short trajectories of length five, the DMN achieved 91.44% accuracy in predicting travel mode, exceeding LSTM and Seq2Seq by 3.41 and 2.28 percentage points respectively. The performance gap widened substantially for longer trajectories.
 

Table 1. Prediction results on the Geolife test set for a future sequence length of 150

Note: Lower mean absolute error (MAE) is better; higher accuracy is better. Sequence length denotes predicted trajectory points.

Wilcoxon signed-rank tests indicated that the improvements over both baselines were statistically significant across all three prediction tasks.

Ablation experiments also clarified why the model performed well. Removing inter-modal attention reduced performance across every task, confirming that explicitly learning the connections between location, time and mode adds predictive value. Meanwhile, single-task versions produced only marginal accuracy gains while requiring substantially more combined training time.

Better forecasts could make transport more responsive

Accurate multi-task forecasts could support several smart transport applications. Traffic management systems could use predicted locations and travel times to improve dynamic routing and adjust signal timings before congestion intensifies. Predicted transport modes could also help authorities coordinate services and synchronise transfers across road and public transport networks.

This study demonstrates that trajectory prediction becomes more effective when location, time and transport mode are treated as interconnected aspects of the same journey. By explicitly modelling these relationships, the DMN improves predictive performance while avoiding the computational duplication of separate models. This approach could provide a valuable foundation for more anticipatory, coordinated and efficient urban transport systems.

Prof. Wang was recognised by Stanford University as one of the top 2% most-cited scientists worldwide (career-long) in the field of logistics and transportation for four consecutive years, from 2022 to 2025. His research focuses on big data in shipping, sustainable maritime transport, urban transport network modelling, green logistics and supply chain management. He has published more than 100 papers in leading journals, including Transportation Research Parts A–E, Transportation Science, Management Science and Operations Research. In 2023, he received the INFORMS Computing Society Harvey J. Greenberg Research Award. He currently serves as Co-Editor-in-Chief of Transportation Research Part E and Editor-in-Chief of Cleaner Logistics and Supply Chain and Communications in Transportation Research.

Prof. Xu was recognised by Stanford University as one of the top 2% most-cited scientists worldwide (single-year) in the field of logistics and transportation for two years, from 2025 to 2026. Her research focuses on the application of operations research and data-driven analytics to intelligent decision-making, human–agent interaction, game-theoretic policy learning and predictive modelling in complex systems. She has published more than 100 papers in leading journals, including Transportation Science, Transportation Research Parts B–E, Reliability Engineering & System Safety, Information Fusion and Computers & Operations Research. Prof. Xu currently serves as an Associate Editor of Transportation Science and Transportation Research Part D, as well as an Editorial Board Editor of Transportation Research Parts C and E.

[1] Da Lei, Min Xu, Shuaian Wang, A deep multimodal network for multi-task trajectory prediction, Information Fusion, Volume 113, 2025, 102597, ISSN 1566-2535, https://doi.org/10.1016/j.inffus.2024.102597. 

[2] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in Neural Information Processing Systems, 30 (2017).

[3] J. Sun, J. Kim, Joint prediction of next location and travel time from urban vehicle trajectories using long short-term memory neural networks, Transportation Research Part C: Emerging Technologies, 128 (2021) 103114.

[4] L. You, S. Xiao, Q. Peng, C. Claramunt, X. Han, Z. Guan, J. Zhang, St-seq2seq: A spatio-temporal feature-optimized seq2seq model for short-term vessel trajectory prediction, IEEE Access 8 (2020) 218565–218574.

	Prof. Hans Shuaian WANG Professor, Department of Logistics and Maritime Studies Deputy Director, PolyU Maritime Data and Sustainable Development Centre
	Prof. Min XU Associate Professor, Department of Industrial and Systems Engineering