Analyzing Deep Learning Model Inferences for Image Classification using OpenVINO

Conference · Wed Jan 01 04:00:00 EST 2020

DOI:https://doi.org/10.1109/IPDPSW50202.2020.00152· OSTI ID:1804060

Jin, Zheming; Finkel, Hal

It may be desirable to execute deep learning model inferences on an integrated GPU at the edge. While such GPUs are much less powerful than discrete GPUs, it is able to deliver higher floating-point operations per second than a CPU located on the same die. For edge devices, the benefit of moving to lower precision with minimal loss of accuracy to obtain higher performance is also attractive. Hence, we chose 14 deep learning models for image classification to evaluate their inference performance with the OpenVINO toolkit. Then, we analyzed the implementation of the fastest inference model of all the models. The experimental results are promising. Compared to the performance of full-precision (FP32) models, the speedup of the 8-bit (INT8) quantization ranges from 1.02 to 1.56 on an Intel (R) Xeon (R) 4-core CPU, and the speedup of the FP16 models ranges from 1.1 to 2 on an Intel (R) Iris (TM) Pro GPU. For the FP32 models, the GPU is on average 1.5X faster than the CPU.

• Research Organization: Argonne National Laboratory (ANL)
• Sponsoring Organization: USDOE Office of Science
• DOE Contract Number: AC02-06CH11357
• OSTI ID: 1804060
• Country of Publication: United States
• Language: English

Related Subjects

CPU
GPU
deep learning
inference
quantization