ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum

Rosendo, Daniel; Mattoso, Marta; Costan, Alexandru; Santos Souza, Renan; Pina, Debora; Valduriez, Patrick; Antoniu, Gabriel

doi:10.1109/CLUSTER52292.2023.00026

ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum

Conference · 01 October 2023

DOI:https://doi.org/10.1109/CLUSTER52292.2023.00026· OSTI ID:2301621

Rosendo, Daniel ^[1]; Mattoso, Marta ^[2]; Costan, Alexandru ^[1]; Santos Souza, Renan ^[3]; Pina, Debora ^[2]; Valduriez, Patrick ^[4]; Antoniu, Gabriel ^[1]

University of Rennes, Inria, CNRS, IRISA
Federal University of Rio de Janeiro
ORNL
University of Montpellier, Inria, CNRS, LIRMM

Modern scientific workflows require hybrid infrastructures combining numerous decentralized resources on the IoT/Edge interconnected to Cloud/HPC systems (aka the Computing Continuum) to enable their optimized execution. Understanding and optimizing the performance of such complex Edge-to-Cloud workflows is challenging. Capturing the provenance of key performance indicators, with their related data and processes, may assist in understanding and optimizing workflow executions. However, the capture overhead can be prohibitive, particularly in resource-constrained devices, such as the ones on the IoT/Edge.To address this challenge, based on a performance analysis of existing systems, we propose ProvLight, a tool to enable efficient provenance capture on the IoT/Edge. We leverage simplified data models, data compression and grouping, and lightweight transmission protocols to reduce overheads. We further integrate ProvLight into the E2Clab framework to enable workflow provenance capture across the Edge-to-Cloud Continuum. This integration makes E2Clab a promising platform for the performance optimization of applications through reproducible experiments.We validate ProvLight at a large scale with synthetic workloads on 64 real-life IoT/Edge devices in the FIT IoT LAB testbed. Evaluations show that ProvLight outperforms state-of-the-art systems like ProvLake and DfAnalyzer in resource-constrained devices. ProvLight is 26—37x faster to capture and transmit provenance data; uses 5—7x less CPU; 2x less memory; transmits 2x less data; and consumes 2—2.5x less energy. ProvLight [1] and E2Clab [2] are available as open-source tools.

View Conference

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 2301621

Resource Relation:: Conference: IEEE CLUSTER 2023: IEEE International Conference on Cluster Computing - Santa Fe, New Mexico, United States of America - 10/31/2023 8:00:00 AM-11/3/2023 8:00:00 AM

Country of Publication:: United States

Language:: English

References (40)

Keeping track of user steering actions in dynamic workflows Souza, Renan; Silva, Vítor; Camata, Jose J. Future Generation Computer Systems, Vol. 99 https://doi.org/10.1016/j.future.2019.05.011	journal	October 2019
A Comparative Evaluation of AMQP, MQTT and HTTP Protocols Using Real-Time Public Smart City Data Gemirter, Cavide Balki; Senturca, Cagatay; Baydere, Sebnem 2021 6th International Conference on Computer Science and Engineering (UBMK) https://doi.org/10.1109/UBMK52708.2021.9559032	conference	September 2021
Provenance Supporting Hyperparameter Analysis in Deep Neural Networks Pina, Débora; Kunstmann, Liliane; de Oliveira, Daniel Lecture Notes in Computer Science https://doi.org/10.1007/978-3-030-80960-7_2	book	January 2021
Comparison with HTTP and MQTT In Internet of Things (IoT) Wukkadada, Bharati; Wankhede, Kirti; Nambiar, Ramith 2018 International Conference on Inventive Research in Computing Applications (ICIRCA) https://doi.org/10.1109/ICIRCA.2018.8597401	conference	July 2018
Encyclopedia of Database Systems Liu, Ling; Özsu, M. Tamer https://doi.org/10.1007/978-0-387-39940-9		January 2009
Coding the Computing Continuum: Fluid Function Execution in Heterogeneous Computing Environments Kumar, Rohan; Baughman, Matt; Chard, Ryan 2021 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) https://doi.org/10.1109/IPDPSW52791.2021.00018	conference	June 2021
Efficient Runtime Capture of Multiworkflow Data Using Provenance Souza, Renan; Azevedo, Leonardo; Thiago, Raphael 2019 15th International Conference on eScience (eScience) https://doi.org/10.1109/eScience.2019.00047	conference	September 2019
A survey on provenance: What for? What form? What from? Herschel, Melanie; Diestelkämper, Ralf; Ben Lahmar, Houssem The VLDB Journal, Vol. 26, Issue 6 https://doi.org/10.1007/s00778-017-0486-1	journal	October 2017
The Next 5 Years: What Opportunities Should the Database Community Seize to Maximize its Impact? Balazinska, Magda; Chaudhuri, Surajit; Ailamaki, Anastasia Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data https://doi.org/10.1145/3318464.3394837	conference	May 2020
DfAnalyzer: Runtime dataflow analysis tool for Computational Science and Engineering applications Silva, Vítor; Campos, Vinícius; Guedes, Thaylon SoftwareX, Vol. 12 https://doi.org/10.1016/j.softx.2020.100592	journal	July 2020
An On-Device Federated Learning Approach for Cooperative Model Update Between Edge Devices Ito, Rei; Tsukada, Mineto; Matsutani, Hiroki IEEE Access, Vol. 9 https://doi.org/10.1109/ACCESS.2021.3093382	journal	January 2021
Edge-Assisted Rendering of 360° Videos Streamed to Head-Mounted Virtual Reality Lo, Wen-Chih; Huang, Chih-Yuan; Hsu, Cheng-Hsin 2018 IEEE International Symposium on Multimedia (ISM) https://doi.org/10.1109/ISM.2018.00016	conference	December 2018
IoT-Enabled Smart Energy Grid: Applications and Challenges Abir, S. M. Abu Adnan; Anwar, Adnan; Choi, Jinho IEEE Access, Vol. 9 https://doi.org/10.1109/ACCESS.2021.3067331	journal	January 2021
Deep Anomaly Detection for Time-Series Data in Industrial IoT: A Communication-Efficient On-Device Federated Learning Approach Liu, Yi; Garg, Sahil; Nie, Jiangtian IEEE Internet of Things Journal, Vol. 8, Issue 8 https://doi.org/10.1109/JIOT.2020.3011726	journal	April 2021
Federated Learning for Internet of Things Zhang, Tuo; He, Chaoyang; Ma, Tianhao Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems https://doi.org/10.1145/3485730.3493444	conference	November 2021
Automating Edge-to-cloud Workflows for Science: Traversing the Edge-to-cloud Continuum with Pegasus Tanaka, Ryan; Papadimitriou, George; Viswanath, Sai Charan 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid) https://doi.org/10.1109/CCGrid54584.2022.00098	conference	May 2022
Combining Heuristics to Optimize and Scale the Placement of IoT Applications in the Fog Xia, Ye; Etchevers, Xavier; Letondeur, Loic 2018 IEEE/ACM 11th International Conference on Utility and Cloud Computing (UCC) https://doi.org/10.1109/UCC.2018.00024	conference	December 2018
Reproducible Performance Optimization of Complex Applications on the Edge-to-Cloud Continuum Rosendo, Daniel; Costan, Alexandru; Antoniu, Gabriel 2021 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/Cluster48925.2021.00043	conference	September 2021
Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence Deng, Shuiguang; Zhao, Hailiang; Fang, Weijia IEEE Internet of Things Journal, Vol. 7, Issue 8 https://doi.org/10.1109/JIOT.2020.2984887	journal	August 2020
Distributed intelligence on the Edge-to-Cloud Continuum: A systematic literature review Rosendo, Daniel; Costan, Alexandru; Valduriez, Patrick Journal of Parallel and Distributed Computing, Vol. 166 https://doi.org/10.1016/j.jpdc.2022.04.004	journal	August 2022
Big data and extreme-scale computing: Pathways to Convergence-Toward a shaping strategy for a future software and data ecosystem for scientific inquiry Asch, M.; Moore, T.; Badia, R. The International Journal of High Performance Computing Applications, Vol. 32, Issue 4 https://doi.org/10.1177/1094342018778123	journal	July 2018
The trade-offs between Fog Processing and Communications in latency-sensitive Vehicular Fog Computing de Mendonça, Francisco Ferreira; Kokkinogenis, Zafeiris; Dias, Kelvin Lopes Pervasive and Mobile Computing, Vol. 84 https://doi.org/10.1016/j.pmcj.2022.101638	journal	August 2022
E2Clab: Exploring the Computing Continuum through Repeatable, Replicable and Reproducible Edge-to-Cloud Experiments Rosendo, Daniel; Silva, Pedro; Simonin, Matthieu 2020 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER49012.2020.00028	conference	September 2020
EnosLib: A Library for Experiment-Driven Research in Distributed Computing Cherrueau, Ronan-Alexandre; Delavergne, Marie; van Kempen, Alexandre IEEE Transactions on Parallel and Distributed Systems, Vol. 33, Issue 6 https://doi.org/10.1109/TPDS.2021.3111159	journal	June 2022
EdgeFed: Optimized Federated Learning Based on Edge Computing Ye, Yunfan; Li, Shen; Liu, Fang IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2020.3038287	journal	January 2020
Runtime Analysis of Whole-System Provenance Pasquier, Thomas; Han, Xueyuan; Moyer, Thomas CCS '18: 2018 ACM SIGSAC Conference on Computer and Communications Security, Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security https://doi.org/10.1145/3243734.3243776	conference	October 2018
LineageChain: a fine-grained, secure and efficient data provenance system for blockchains Ruan, Pingcheng; Dinh, Tien Tuan Anh; Lin, Qian The VLDB Journal, Vol. 30, Issue 1 https://doi.org/10.1007/s00778-020-00646-1	journal	January 2021
Genoma: Distributed Provenance as a Service for IoT-based Systems Narendra, Nanjangud C.; Shukla, Anshu; Nayak, Sambit 2019 IEEE 5th World Forum on Internet of Things (WF-IoT) https://doi.org/10.1109/WF-IoT.2019.8767335	conference	April 2019
Grid'5000: A Large Scale And Highly Reconfigurable Experimental Grid Testbed Bolze, Raphaël; Cappello, Franck; Caron, Eddy The International Journal of High Performance Computing Applications, Vol. 20, Issue 4 https://doi.org/10.1177/1094342006070078	journal	November 2006
Managing Large-Scale Workflow Execution from Resource Provisioning to Provenance Tracking: The CyberShake Example Deelman, Ewa; Callaghan, Scott; Field, Edward 2006 Second IEEE International Conference on e-Science and Grid Computing (e-Science'06) https://doi.org/10.1109/E-SCIENCE.2006.261098	conference	December 2006
FIT IoT-LAB: A large scale open experimental IoT testbed Adjih, Cedric; Baccelli, Emmanuel; Fleury, Eric 2015 IEEE 2nd World Forum on Internet of Things (WF-IoT) https://doi.org/10.1109/WF-IoT.2015.7389098	conference	December 2015
Taverna: lessons in creating a workflow environment for the life sciences Oinn, Tom; Greenwood, Mark; Addis, Matthew Concurrency and Computation: Practice and Experience, Vol. 18, Issue 10 https://doi.org/10.1002/cpe.993	journal	January 2006
Active Provenance for Data-Intensive Workflows: Engaging Users and Developers Spinuso, Alessandro; Atkinson, Malcolm; Magnoni, Federica 2019 15th International Conference on eScience (eScience) https://doi.org/10.1109/eScience.2019.00077	conference	September 2019
Towards a provenance collection framework for Internet of Things devices Nwafor, Ebelechukwu; Campbell, Andre; Hill, David 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) https://doi.org/10.1109/UIC-ATC.2017.8397531	conference	August 2017
An Approach to Standalone Provenance Systems for Big Social Provenance Data Tas, Yucel; Baeth, Mohamed Jehad; Aktas, Mehmet S. 2016 12th International Conference on Semantics, Knowledge and Grids (SKG) https://doi.org/10.1109/SKG.2016.010	conference	August 2016
Scientific workflow management and the Kepler system Ludäscher, Bertram; Altintas, Ilkay; Berkley, Chad Concurrency and Computation: Practice and Experience, Vol. 18, Issue 10 https://doi.org/10.1002/cpe.994	journal	January 2006
Prov-Io Han, Runzhou; Byna, Suren; Tang, Houjun Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/3502181.3531477	conference	June 2022
Pegasus, a workflow management system for science automation Deelman, Ewa; Vahi, Karan; Juve, Gideon Future Generation Computer Systems, Vol. 46 https://doi.org/10.1016/j.future.2014.10.008	journal	May 2015
Survey on the Analysis of User Interactions and Visualization Provenance Xu, Kai; Ottley, Alvitta; Walchshofer, Conny Computer Graphics Forum, Vol. 39, Issue 3 https://doi.org/10.1111/cgf.14035	journal	June 2020
The W3C PROV family of specifications for modelling provenance metadata Missier, Paolo; Belhajjame, Khalid; Cheney, James Proceedings of the 16th International Conference on Extending Database Technology https://doi.org/10.1145/2452376.2452478	conference	March 2013

Similar Records

Towards Lightweight Data Integration Using Multi-Workflow Provenance and Data Observability

Conference · 2023 · OSTI ID:2301623

Santos Souza, Renan; Skluzacek, Tyler; Wilkinson, Sean; +2 more

Workflow Provenance in the Computing Continuum for Responsible, Trustworthy, and Energy-Efficient AI

Conference · 2024 · OSTI ID:2478344

Santos Souza, Renan; Caino-Lores, Silvina; Coletti, Mark; +5 more

NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning

Conference · 2022 · OSTI ID:1890739

Andersen, Aaron; Rager, David

ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum

Citation Formats

References (40)

Similar Records

Related Subjects