Skip to main content
U.S. Department of Energy
Office of Scientific and Technical Information

Big data analytics in smart grids: state-of-the-art, challenges, opportunities, and future directions

Journal Article · · IET Smart Grid
 [1];  [2];  [3];  [3];  [4];  [5];  [3];  [3];  [6];  [7];  [8];  [9];  [10]
  1. Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
  2. Michigan Technological Univ., Houghton, MI (United States)
  3. Idaho National Lab. (INL), Idaho Falls, ID (United States)
  4. GE Grid Solutions, Redmond, WA (United States)
  5. South Dakota State Univ., Brookings, SD (United States)
  6. IBM Research, Almaden, CA (United States)
  7. Oncor Electric Delivery, Dallas, TX (United States)
  8. Virginia Commonwealth Univ., Richmond, VA (United States)
  9. Independent System Operator New England, Holyoke, MA (United States)
  10. California Independent System Operator, Folsom, CA (United States)

Big data has a potential to unlock novel groundbreaking opportunities in the power grid sector that enhances a multitude of technical, social, and economic gains. The currently untapped potential of applying the science of big data for better planning and operation of the power grid is a very challenging task and needs significant efforts all-around. As power grid technologies evolve in conjunction with measurement and communication technologies, this results in unprecedented amount of heterogeneous big data sets from diverse sources. In particular, computational complexity, data security, and operational integration of big data into utility decision frameworks are the key challenges to transform the heterogeneous large dataset into actionable outcomes. Moreover, due to the complex nature of power grids along with the need to balance power in real time, seamless integration of big data into utility operations is very critical. In this context, big data analytics combined with grid visualization can lead to better situational awareness and predictive decisions. This paper presents a comprehensive state-of-the-art review of big data analytics and its applications in power grids, and also identifies challenges and opportunities from utility, industry, and research perspectives. The paper analyzes research gaps and presents insights on future research directions to integrate big data analytics into electric utility decision framework. Detailed information for utilities looking to apply big data analytics and details insights on how utilities can enhance revenue streams and bring disruptive innovation in the industry are discussed. More importantly, general guidelines for utilities to make the right investment in the adoption of big data analytics by unveiling interdependencies among critical infrastructures and operations are also provided.

Research Organization:
Idaho National Laboratory (INL), Idaho Falls, ID (United States)
Sponsoring Organization:
USDOE Office of Nuclear Energy (NE)
Grant/Contract Number:
AC07-05ID14517
OSTI ID:
1559939
Alternate ID(s):
OSTI ID: 1639296
Report Number(s):
INL/JOU--17-42462-Rev000
Journal Information:
IET Smart Grid, Journal Name: IET Smart Grid Journal Issue: 2 Vol. 2; ISSN 2515-2947
Publisher:
The Institution of Engineering and TechnologyCopyright Statement
Country of Publication:
United States
Language:
English

References (132)

Quantifying uncertainties of neural network-based electricity price forecasts journal December 2013
Big Data Analytics for Dynamic Energy Management in Smart Grids journal September 2015
Cloud Computing for Smart Grid applications journal September 2014
State-of-the-art, challenges, and future trends in security constrained optimal power flow journal August 2011
Improving data partition schemes in Smart Grids via clustering data streams journal October 2014
A review of electric load classification in smart grid environment journal August 2013
Smart power grid and cloud computing journal August 2013
Demand side management using artificial neural networks in a smart grid environment journal January 2015
Big data driven smart energy management: From big data to big insights journal April 2016
The contributions of cloud technologies to smart grid journal June 2016
Power Systems of the Future journal March 2015
Cybersecurity for distributed energy resources and smart inverters journal December 2016
ARCADES: analysis of risk from cyberattack against defensive strategies for the power grid journal September 2018
Vulnerable transmission line identification using ISH theory in power grids journal January 2018
Validation of a PMU-based fault location identification method for smart distribution network with photovoltaics using real-time data journal November 2018
Computationally and statistically efficient model fitting techniques journal May 2016
A Quadtree-Based Dynamic Attribute Indexing Method journal March 1998
Big Data Deep Learning: Challenges and Perspectives journal January 2014
High-Performance Extreme Learning Machines: A Complete Toolbox for Big Data Applications journal January 2015
Impact of Interdisciplinary Research on Planning, Running, and Managing Electromobility as a Smart Grid Extension journal January 2015
Understanding Customer Behavior in Multi-Tier Demand Response Management Program journal January 2015
Internet of Things and Big Data Analytics for Smart and Connected Communities journal January 2016
Energy big data: A survey journal January 2016
Open Data-Set of Seven Canadian Cities journal January 2017
Live Data Analytics With Collaborative Edge and Cloud Processing in Wireless IoT Networks journal January 2017
Mobile Cloud-Based Big Healthcare Data Processing in Smart Cities journal January 2017
A New Method for Time-Series Big Data Effective Storage journal January 2017
Multiple Power Line Outage Detection in Smart Grids: Probabilistic Bayesian Approach journal January 2018
Mining Human Activity Patterns From Smart Home Big Data for Health Care Applications journal January 2017
Analyzing Cascading Failures in Smart Grids under Random and Targeted Attacks conference May 2014
Demand response targeting using big data analytics conference October 2013
Big data analytics for demand response: Clustering over space and time conference October 2015
Towards a Semantic Extract-Transform-Load (ETL) Framework for Big Data Integration conference June 2014
A reliable data aggregation mechanism with Homomorphic Encryption in Smart Grid AMI networks conference January 2016
An experimental approach towards big data for analyzing memory utilization on a hadoop cluster using HDFS and MapReduce
  • Pal, Amrit; Agrawal, Sanjay
  • 2014 International Conference on Networks & Soft Computing (ICNSC), 2014 First International Conference on Networks & Soft Computing (ICNSC2014) https://doi.org/10.1109/CNSC.2014.6906718
conference August 2014
Power Grids of the Future: Why Smart Means Complex conference February 2010
Correlation Analysis of Big Data to Support Machine Learning
  • Pandey, Rajiv; Dhoundiyal, Manoj; Kumar, Amrendra
  • 2015 Fifth International Conference on Communication Systems and Network Technologies (CSNT) https://doi.org/10.1109/CSNT.2015.32
conference April 2015
Cyber Physical Security Analytics for Anomalies in Transmission Protection Systems
  • Ahmed, A.; Krishnan, V. V. G.; Foroutan, S. A.
  • 2018 IEEE Industry Applications Society Annual Meeting (IAS2018), 2018 IEEE Industry Applications Society Annual Meeting (IAS) https://doi.org/10.1109/IAS.2018.8544672
conference September 2018
Distributed data management in energy sector using Hadoop conference September 2015
A cloud-based architecture for Big-Data analytics in smart grid: A proposal conference December 2013
Towards Service-Oriented Middleware for Fog and Cloud Integrated Cyber Physical Systems conference June 2017
Data Security and Encryption Technology Research on Smart Grid Communication System conference March 2016
Benchmarking Streaming Computation Engines: Storm, Flink and Spark Streaming conference May 2016
The role of big data in improving power system operation and protection
  • Kezunovic, Mladen; Xie, Le; Grijalva, Santiago
  • 2013 IREP Symposium - Bulk Power System Dynamics and Control - IX Optimization, Security and Control of the Emerging Power Grid (IREP), 2013 IREP Symposium Bulk Power System Dynamics and Control - IX Optimization, Security and Control of the Emerging Power Grid https://doi.org/10.1109/IREP.2013.6629368
conference August 2013
Automated analysis of power systems disturbance records: Smart Grid big data perspective conference May 2014
Big data analytics in power distribution systems conference February 2015
An Ingestion and Analytics Architecture for IoT Applied to Smart City Use Cases journal April 2018
A Two-Way Street: Green Big Data Processing for a Greener Smart Grid journal June 2017
Graph Algorithms for Topology Identification Using Power Grid Probing journal October 2018
A secure decentralized data-centric information infrastructure for smart grid journal November 2010
Mobile Edge Computing for Big-Data-Enabled Electric Vehicle Charging journal March 2018
Cloud-Based Software Platform for Big Data Analytics in Smart Grids journal July 2013
Virtual Edge-Based Smart Community Network Management journal November 2016
Leveraging Big Data Analytics to Reduce Healthcare Costs journal November 2013
Big Data and Big Money: The Role of Data in the Financial Sector journal January 2017
Getting Smart journal March 2008
Enabling the Integrated Grid: Leveraging Data to Integrate Distributed Resources and Customers journal January 2016
Big Data Analytics: Making the Smart Grid Smarter [Guest Editorial] journal May 2018
Edge Analytics in the Internet of Things journal April 2015
The Hadoop Distributed File System conference May 2010
Real Time Dynamics Monitoring System (RTDMS®) for use with SynchroPhasor technology in power systems conference July 2011
Addressing the challenges for integrating micro-synchrophasor data with operational system applications conference July 2014
Mosaik: A framework for modular simulation of active components in Smart Grids conference October 2011
Privacy for Smart Meters: Towards Undetectable Appliance Load Signatures
  • Kalogridis, Georgios; Efthymiou, Costas; Denic, Stojan Z.
  • 2010 1st IEEE International Conference on Smart Grid Communications (SmartGridComm), 2010 First IEEE International Conference on Smart Grid Communications https://doi.org/10.1109/SMARTGRID.2010.5622047
conference October 2010
Secure Information Aggregation for Smart Grids Using Homomorphic Encryption
  • Li, Fengjun; Luo, Bo; Liu, Peng
  • 2010 1st IEEE International Conference on Smart Grid Communications (SmartGridComm), 2010 First IEEE International Conference on Smart Grid Communications https://doi.org/10.1109/SMARTGRID.2010.5622064
conference October 2010
Smart Grid Data Cloud: A Model for Utilizing Cloud Computing in the Smart Grid Domain conference October 2010
Petuum: A New Platform for Distributed Machine Learning on Big Data journal June 2015
JouleMR: Towards Cost-Effective and Green-Aware Data Processing Frameworks journal June 2018
Massive Streaming PMU Data Modelling and Analytics in Smart Grid State Evaluation based on Multiple High-Dimensional Covariance Test journal March 2018
Online dimension reduction of synchrophasor data conference May 2012
Compressive Sensing-Based Topology Identification for Smart Grids journal April 2016
Distribution Line Parameter Estimation Under Consideration of Measurement Tolerances journal April 2016
Decision Trees for Uncertain Data journal January 2011
A Robust Regularization Path Algorithm for $\nu $ -Support Vector Classification journal May 2017
Energy Conscious Scheduling for Distributed Computing Systems under Different Operating Conditions journal August 2011
Cloud Computing Applications for Smart Grid: A Survey journal May 2015
Fault Detection and Faulted Line Identification in Active Distribution Networks Using Synchrophasors-Based Real-Time State Estimation journal February 2017
Identification of Umbrella Constraints in DC-Based Security-Constrained Optimal Power Flow journal November 2013
Dimensionality Reduction of Synchrophasor Data for Early Event Detection: Linearized Analysis journal November 2014
Outage Management of Distribution Systems Incorporating Information From Smart Meters journal September 2016
Sparse and Redundant Representation-Based Smart Meter Data Compression and Pattern Extraction journal May 2017
Distributed Energy Resources Topology Identification via Graphical Modeling journal July 2017
Smart Transmission Grid Applications and Their Supporting Infrastructure journal June 2010
The Next Generation of Power Distribution Systems journal December 2010
Smart Grid Technologies for Autonomous Operation and Control journal March 2011
Data Mining for Electricity Price Classification and the Application to Demand-Side Management journal June 2012
Aggregated-Proofs Based Privacy-Preserving Authentication for V2G Networks in the Smart Grid journal December 2012
Front-End Electronic Circuit Topology Analysis for Model-Driven Classification and Monitoring of Appliance Loads in Smart Buildings journal December 2012
Dependable Demand Response Management in the Smart Grid: A Stackelberg Game Approach journal March 2013
Applications and Trends of High Performance Computing for Electric Power Systems: Focusing on Smart Grid journal June 2013
Energy and Performance Management of Green Data Centers: A Profit Maximization Approach journal June 2013
Using a Distributed Agent-Based Communication Enabled Special Protection System to Enhance Smart Grid Security journal June 2013
Smart Grid Data Integrity Attacks journal September 2013
Clustering-Based Improvement of Nonparametric Functional Time Series Forecasting: Application to Intra-Day Household-Level Load Curves journal January 2014
CPIndex: Cyber-Physical Vulnerability Assessment for Power-Grid Infrastructures journal March 2015
Segmentation and Classification of Commercial Building Occupants by Energy-Use Efficiency and Predictability journal May 2015
Electricity Theft Detection in AMI Using Customers’ Consumption Patterns journal January 2016
Power System Real-Time Monitoring by Using PMU-Based Robust State Estimation Method journal January 2016
Data Compression in Smart Distribution Systems via Singular Value Decomposition journal January 2017
Multivariable Grid Admittance Identification for Impedance Stabilization of Active Distribution Networks journal May 2017
A Data-Driven Approach for Estimating the Power Generation of Invisible Solar Sites journal September 2016
Distributed Data Analytics Platform for Wide-Area Synchrophasor Measurement Systems journal September 2016
Distribution System Model Calibration With Big Data From AMI and PV Inverters journal September 2016
Estimating Power Generation of Invisible Solar Sites Using Publicly Available Data journal September 2016
A Novel Dominant Mode Estimation Method for Analyzing Inter-Area Oscillation in China Southern Power Grid journal September 2016
Smart Metering Load Data Compression Based on Load Feature Identification journal September 2016
Data-Driven Power Outage Detection by Social Sensors journal September 2016
Clustering of Electricity Consumption Behavior Dynamics Toward Big Data Applications journal September 2016
Power System Transient Stability Assessment Based on Big Data and the Core Vector Machine journal September 2016
Spatial-Temporal Synchrophasor Data Characterization and Analytics in Smart Grid Fault Detection, Identification, and Impact Causal Analysis journal September 2016
A Data-Driven Approach for Detection and Estimation of Residential PV Installations journal September 2016
A Novel Association Rule Mining Method of Big Data for Power Transformers State Parameters Based on Probabilistic Graph Model journal March 2018
Energy Big Data Analytics and Security: Challenges and Opportunities journal September 2016
Cloud Computing for Power System Simulations at ISO New England—Experiences and Challenges journal November 2016
TAI: A Threshold-Based Anonymous Identification Scheme for Demand-Response in Smart Grids journal July 2018
Identifying Topology of Low Voltage Distribution Networks Based on Smart Meter Data journal September 2018
Partial Knowledge Data-Driven Event Detection for Power Distribution Networks journal September 2018
Deep Learning for Household Load Forecasting—A Novel Pooling Deep RNN journal September 2018
A Hierarchical Framework for Smart Grid Anomaly Detection Using Large-Scale Smart Meter Data journal November 2018
Efficient and Privacy-Preserving Data Aggregation Scheme for Smart Grid Against Internal Adversaries journal September 2017
AMI-Enabled Distribution Network Line Outage Identification via Multi-Label SVM journal September 2018
Identifying Optimal Energy Flow Solvability in Electricity-Gas Integrated Energy Systems journal April 2017
A Data-Centric Storage Approach for Efficient Query of Large-Scale Smart Grid
  • Wang, Yan; Deng, Qingxu; Liu, Wei
  • 2012 9th Web Information Systems and Applications Conference (WISA), 2012 Ninth Web Information Systems and Applications Conference https://doi.org/10.1109/WISA.2012.27
conference November 2012
Differentially private aggregation of distributed time-series with transformation and encryption
  • Rastogi, Vibhor; Nath, Suman
  • SIGMOD/PODS '10: International Conference on Management of Data, Proceedings of the 2010 ACM SIGMOD International Conference on Management of data https://doi.org/10.1145/1807167.1807247
conference June 2010
Cloud Computing: Survey on Energy Efficiency journal January 2015
Quadtree and R-tree indexes in oracle spatial: a comparison using GIS data
  • Kothuri, Ravi Kanth V.; Ravada, Siva; Abugov, Daniel
  • Proceedings of the 2002 ACM SIGMOD international conference on Management of data - SIGMOD '02 https://doi.org/10.1145/564691.564755
conference January 2002
Evaluating probabilistic queries over imprecise data
  • Cheng, Reynold; Kalashnikov, Dmitri V.; Prabhakar, Sunil
  • Proceedings of the 2003 ACM SIGMOD international conference on on Management of data - SIGMOD '03 https://doi.org/10.1145/872757.872823
conference January 2003
Big Data Platforms and Techniques journal January 2016
A survey on platforms for big data analytics journal October 2014
Computational Intelligence Approaches for Energy Load Forecasting in Smart Energy Management Grids: State of the Art, Future Challenges, and Research Directions journal March 2018
Big Data Analytics for Discovering Electricity Consumption Patterns in Smart Cities journal March 2018
Electricity Consumption Clustering Using Smart Meter Data journal April 2018

Cited By (3)

Bounding Regression Errors in Data-Driven Power Grid Steady-State Models journal March 2021
Design of Experiments in the Methodology for Interoperability Testing: Evaluating AMI Message Exchange journal March 2019
A Novel on Transmission Line Tower Big Data Analysis Model Using Altered K-means and ADQL journal June 2019