Privacy-Preserving Real-Time Action Detection in Intelligent Vehicles Using Federated Learning-Based Temporal Recurrent Network

Gökcen, Alpaslan; Boyacı, Ali

doi:10.3390/electronics13142820

Privacy-Preserving Real-Time Action Detection in Intelligent Vehicles Using Federated Learning-Based Temporal Recurrent Network

Journal Article · Thu Jul 18 00:00:00 EDT 2024 · Electronics

DOI:https://doi.org/10.3390/electronics13142820· OSTI ID:2438866

^[1]; ^[2]

Istanbul Commerce University (Turkey); Turkcell İletişim Hizmetleri, Istanbul (Turkey)
Istanbul Commerce University (Turkey); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

This study introduces a privacy-preserving approach for the real-time action detection in intelligent vehicles using a federated learning (FL)-based temporal recurrent network (TRN). This approach enables edge devices to independently train models, enhancing data privacy and scalability by eliminating central data consolidation. Our FL-based TRN effectively captures temporal dependencies, anticipating future actions with high precision. Extensive testing on the Honda HDD and TVSeries datasets demonstrated robust performance in centralized and decentralized settings, with competitive mean average precision (mAP) scores. The experimental results highlighted that our FL-based TRN achieved an mAP of 40.0% in decentralized settings, closely matching the 40.1% in centralized configurations. Notably, the model excelled in detecting complex driving maneuvers, with mAPs of 80.7% for intersection passing and 78.1% for right turns. These outcomes affirm the model’s accuracy in action localization and identification. The system showed significant scalability and adaptability, maintaining robust performance across increased client device counts. The integration of a temporal decoder enabled predictions of future actions up to 2 s ahead, enhancing the responsiveness. Our research advances intelligent vehicle technology, promoting safety and efficiency while maintaining strict privacy standards.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 2438866

Journal Information:: Electronics, Journal Name: Electronics Journal Issue: 14 Vol. 13; ISSN 2079-9292

Publisher:: MDPICopyright Statement

Country of Publication:: United States

Language:: English

References (26)

Privacy-preserving collaborative AI for distributed deep learning with cross-sectional data Iqbal, Saeed; Qureshi, Adnan N.; Alhussein, Musaed Multimedia Tools and Applications https://doi.org/10.1007/s11042-023-17202-y	journal	November 2023
Dense Trajectories and Motion Boundary Descriptors for Action Recognition Wang, Heng; Kläser, Alexander; Schmid, Cordelia International Journal of Computer Vision, Vol. 103, Issue 1 https://doi.org/10.1007/s11263-012-0594-8	journal	March 2013
A survey of federated learning for edge computing: Research problems and solutions Xia, Qi; Ye, Winson; Tao, Zeyi High-Confidence Computing, Vol. 1, Issue 1 https://doi.org/10.1016/j.hcc.2021.100008	journal	June 2021
A federated learning system with enhanced feature extraction for human activity recognition Xiao, Zhiwen; Xu, Xin; Xing, Huanlai Knowledge-Based Systems, Vol. 229 https://doi.org/10.1016/j.knosys.2021.107338	journal	October 2021
Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network Sherstinsky, Alex Physica D: Nonlinear Phenomena, Vol. 404 https://doi.org/10.1016/j.physd.2019.132306	journal	March 2020
Learning realistic human actions from movies Laptev, Ivan; Marszalek, Marcin; Schmid, Cordelia 2008 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2008.4587756	conference	June 2008
ImageNet: A large-scale hierarchical image database Deng, Jia; Dong, Wei; Socher, Richard 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), 2009 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2009.5206848	conference	June 2009
Fast unsupervised ego-action learning for first-person sports videos Kitani, Kris M.; Okabe, Takahiro; Sato, Yoichi CVPR 2011 https://doi.org/10.1109/CVPR.2011.5995406	conference	June 2011
Action recognition by dense trajectories Wang, Heng; Klaser, Alexander; Schmid, Cordelia CVPR 2011 https://doi.org/10.1109/CVPR.2011.5995407	conference	June 2011
Delving into egocentric actions Li, Yin; Rehg, James M. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2015.7298625	conference	June 2015
Long-term recurrent convolutional networks for visual recognition and description Donahue, Jeff; Hendricks, Lisa Anne; Guadarrama, Sergio 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2015.7298878	conference	June 2015
Going Deeper into First-Person Activity Recognition Ma, Minghuang; Fan, Haoqi; Kitani, Kris M. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.209	conference	June 2016
Temporal Convolutional Networks for Action Segmentation and Detection Lea, Colin; Flynn, Michael D.; Vidal, Rene 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.113	conference	July 2017
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset Carreira, Joao; Zisserman, Andrew 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2017.502	conference	July 2017
Real-World Anomaly Detection in Surveillance Videos Sultani, Waqas; Chen, Chen; Shah, Mubarak 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00678	conference	June 2018
Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning Ramanishka, Vasili; Chen, Yi-Ting; Misu, Teruhisa 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00803	conference	June 2018
Federated Learning-based Driver Activity Recognition for Edge Devices Doshi, Keval; Yilmaz, Yasin 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) https://doi.org/10.1109/CVPRW56347.2022.00377	conference	June 2022
Learning Spatiotemporal Features with 3D Convolutional Networks Tran, Du; Bourdev, Lubomir; Fergus, Rob 2015 IEEE International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2015.510	conference	December 2015
Temporal Recurrent Networks for Online Action Detection Xu, Mingze; Gao, Mingfei; Chen, Yi-Ting 2019 IEEE/CVF International Conference on Computer Vision (ICCV) https://doi.org/10.1109/ICCV.2019.00563	conference	October 2019
Modeling Temporal Structure with LSTM for Online Action Detection De Geest, Roeland; Tuytelaars, Tinne 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) https://doi.org/10.1109/WACV.2018.00173	conference	March 2018
Fully-Coupled Two-Stream Spatiotemporal Networks for Extremely Low Resolution Action Recognition Xu, Mingze; Sharghi, Aidean; Chen, Xin 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) https://doi.org/10.1109/WACV.2018.00178	conference	March 2018
FedFSLAR: A Federated Learning Framework for Few-shot Action Recognition Tu, Nguyen Anh; Abu, Assanali; Aikyn, Nartay 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) https://doi.org/10.1109/WACVW60836.2024.00035	conference	January 2024
Long Short-Term Memory Hochreiter, Sepp; Schmidhuber, Jürgen Neural Computation, Vol. 9, Issue 8 https://doi.org/10.1162/neco.1997.9.8.1735	journal	November 1997
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions Alzubaidi, Laith; Zhang, Jinglan; Humaidi, Amjad J. Journal of Big Data, Vol. 8, Issue 1 https://doi.org/10.1186/s40537-021-00444-8	journal	March 2021
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning Szegedy, Christian; Ioffe, Sergey; Vanhoucke, Vincent Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31, Issue 1 https://doi.org/10.1609/aaai.v31i1.11231	journal	February 2017
RED: Reinforced Encoder-Decoder Networks for Action Anticipation Gao, Jiyang; Yang, Zhenheng; Nevatia, Ram Procedings of the British Machine Vision Conference 2017 https://doi.org/10.5244/C.31.92	conference	January 2017

Similar Records

Privacy-preserving federated learning: Application to behind-the-meter solar photovoltaic generation forecasting

Journal Article · Fri Mar 10 19:00:00 EST 2023 · Energy Conversion and Management · OSTI ID:2417657

Adaptive Client Selection in Federated Learning: A Network Anomaly Detection Use Case

Conference · Sun Feb 16 23:00:00 EST 2025 · OSTI ID:2583897

Emerging Technologies for Privacy Preservation in Energy Systems

Conference · Wed Jun 05 00:00:00 EDT 2024 · OSTI ID:2475808

Related Subjects

33 ADVANCED PROPULSION SYSTEMS
data privacy
federated learning
intelligent vehicles
real-time action detection

Privacy-Preserving Real-Time Action Detection in Intelligent Vehicles Using Federated Learning-Based Temporal Recurrent Network

Citation Formats

References (26)

Similar Records

Related Subjects