skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Computationally Efficient Learning of Quality Controlled Word Embeddings for Natural Language Processing

Abstract

Deep learning (DL) has been used for many natural language processing (NLP) tasks due to its superior performance as compared to traditional machine learning approaches. In DL models for NLP, words are represented using word embeddings, which capture both semantic and syntactic information in text. However, 90-95% of the DL trainable parameters are associated with the word embeddings, resulting in a large storage or memory footprint. Therefore, reducing the number of word embedding parameters is critical, especially with the increase of vocabulary size. In this work, we propose a novel approximate word embeddings approach for convolutional neural networks (CNNs) used for text classification tasks. The proposed approach significantly reduces the number of model trainable parameters without noticeably sacrificing in computing performance accuracy. Compared to other techniques, our proposed word embeddings technique does not require modifications to the DL model architecture. We evaluate the performance of the the proposed word embeddings on three classification tasks using two datasets, composed of Yelp and Amazon reviews. The results show that the proposed method can reduce the number of word embeddings parameters by 98% and 99% for the Yelp and Amazon datasets respectively, with no drop in computing accuracy.

Authors:
ORCiD logo [1]; ORCiD logo [1]
  1. ORNL
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1545208
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: IEEE Computer Society Annual Symposium on VLSI - Miami, Florida, United States of America - 7/15/2019 4:00:00 AM-7/17/2019 4:00:00 AM
Country of Publication:
United States
Language:
English

Citation Formats

Alawad, Mohammed M., and Tourassi, Georgia. Computationally Efficient Learning of Quality Controlled Word Embeddings for Natural Language Processing. United States: N. p., 2019. Web.
Alawad, Mohammed M., & Tourassi, Georgia. Computationally Efficient Learning of Quality Controlled Word Embeddings for Natural Language Processing. United States.
Alawad, Mohammed M., and Tourassi, Georgia. Mon . "Computationally Efficient Learning of Quality Controlled Word Embeddings for Natural Language Processing". United States. https://www.osti.gov/servlets/purl/1545208.
@article{osti_1545208,
title = {Computationally Efficient Learning of Quality Controlled Word Embeddings for Natural Language Processing},
author = {Alawad, Mohammed M. and Tourassi, Georgia},
abstractNote = {Deep learning (DL) has been used for many natural language processing (NLP) tasks due to its superior performance as compared to traditional machine learning approaches. In DL models for NLP, words are represented using word embeddings, which capture both semantic and syntactic information in text. However, 90-95% of the DL trainable parameters are associated with the word embeddings, resulting in a large storage or memory footprint. Therefore, reducing the number of word embedding parameters is critical, especially with the increase of vocabulary size. In this work, we propose a novel approximate word embeddings approach for convolutional neural networks (CNNs) used for text classification tasks. The proposed approach significantly reduces the number of model trainable parameters without noticeably sacrificing in computing performance accuracy. Compared to other techniques, our proposed word embeddings technique does not require modifications to the DL model architecture. We evaluate the performance of the the proposed word embeddings on three classification tasks using two datasets, composed of Yelp and Amazon reviews. The results show that the proposed method can reduce the number of word embeddings parameters by 98% and 99% for the Yelp and Amazon datasets respectively, with no drop in computing accuracy.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2019},
month = {7}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: