Distributed Stochastic Gradient Descent Using LDGM Codes

研究成果: Conference contribution

抄録

We consider a distributed learning problem in which the computation is carried out on a system consisting of a master node and multiple worker nodes. In such systems, the existence of slow-running machines called stragglers will cause a significant decrease in performance. Recently, coding theoretic framework, which is named Gradient Coding (GC), for mitigating stragglers in distributed learning has been established by Tandon et al. Most studies on GC are aiming at recovering the gradient information completely assuming that the Gradient Descent (GD) algorithm is used as a learning algorithm. On the other hand, if the Stochastic Gradient Descent (SGD) algorithm is used, it is not necessary to completely recover the gradient information, and its unbiased estimator is sufficient for the learning. In this paper, we propose a distributed SGD scheme using Low Density Generator Matrix (LDGM) codes. In the proposed system, it may take longer time than existing GC methods to recover the gradient information completely, however, it enables the master node to obtain a high-quality unbiased estimator of the gradient at low computational cost and it leads to overall performance improvement.

元の言語English
ホスト出版物のタイトル2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings
出版者Institute of Electrical and Electronics Engineers Inc.
ページ1417-1421
ページ数5
ISBN(電子版)9781538692912
DOI
出版物ステータスPublished - 2019 7
イベント2019 IEEE International Symposium on Information Theory, ISIT 2019 - Paris, France
継続期間: 2019 7 72019 7 12

出版物シリーズ

名前IEEE International Symposium on Information Theory - Proceedings
2019-July
ISSN(印刷物)2157-8095

Conference

Conference2019 IEEE International Symposium on Information Theory, ISIT 2019
France
Paris
期間19/7/719/7/12

Fingerprint

Stochastic Gradient
Gradient Descent
Generator
Gradient
Learning algorithms
Coding
Descent Algorithm
Unbiased estimator
Gradient Algorithm
Vertex of a graph
Costs
Computational Cost
Learning Algorithm
Sufficient
Decrease
Necessary

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Information Systems
  • Modelling and Simulation
  • Applied Mathematics

これを引用

Horii, S., Yoshida, T., Kobayashi, M., & Matsushima, T. (2019). Distributed Stochastic Gradient Descent Using LDGM Codes. : 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings (pp. 1417-1421). [8849580] (IEEE International Symposium on Information Theory - Proceedings; 巻数 2019-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISIT.2019.8849580

Distributed Stochastic Gradient Descent Using LDGM Codes. / Horii, Shunsuke; Yoshida, Takahiro; Kobayashi, Manabu; Matsushima, Toshiyasu.

2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. p. 1417-1421 8849580 (IEEE International Symposium on Information Theory - Proceedings; 巻 2019-July).

研究成果: Conference contribution

Horii, S, Yoshida, T, Kobayashi, M & Matsushima, T 2019, Distributed Stochastic Gradient Descent Using LDGM Codes. : 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings., 8849580, IEEE International Symposium on Information Theory - Proceedings, 巻. 2019-July, Institute of Electrical and Electronics Engineers Inc., pp. 1417-1421, 2019 IEEE International Symposium on Information Theory, ISIT 2019, Paris, France, 19/7/7. https://doi.org/10.1109/ISIT.2019.8849580
Horii S, Yoshida T, Kobayashi M, Matsushima T. Distributed Stochastic Gradient Descent Using LDGM Codes. : 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2019. p. 1417-1421. 8849580. (IEEE International Symposium on Information Theory - Proceedings). https://doi.org/10.1109/ISIT.2019.8849580
Horii, Shunsuke ; Yoshida, Takahiro ; Kobayashi, Manabu ; Matsushima, Toshiyasu. / Distributed Stochastic Gradient Descent Using LDGM Codes. 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 1417-1421 (IEEE International Symposium on Information Theory - Proceedings).
@inproceedings{31d04ddb176b4361ab421dbf17832ca5,
title = "Distributed Stochastic Gradient Descent Using LDGM Codes",
abstract = "We consider a distributed learning problem in which the computation is carried out on a system consisting of a master node and multiple worker nodes. In such systems, the existence of slow-running machines called stragglers will cause a significant decrease in performance. Recently, coding theoretic framework, which is named Gradient Coding (GC), for mitigating stragglers in distributed learning has been established by Tandon et al. Most studies on GC are aiming at recovering the gradient information completely assuming that the Gradient Descent (GD) algorithm is used as a learning algorithm. On the other hand, if the Stochastic Gradient Descent (SGD) algorithm is used, it is not necessary to completely recover the gradient information, and its unbiased estimator is sufficient for the learning. In this paper, we propose a distributed SGD scheme using Low Density Generator Matrix (LDGM) codes. In the proposed system, it may take longer time than existing GC methods to recover the gradient information completely, however, it enables the master node to obtain a high-quality unbiased estimator of the gradient at low computational cost and it leads to overall performance improvement.",
author = "Shunsuke Horii and Takahiro Yoshida and Manabu Kobayashi and Toshiyasu Matsushima",
year = "2019",
month = "7",
doi = "10.1109/ISIT.2019.8849580",
language = "English",
series = "IEEE International Symposium on Information Theory - Proceedings",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "1417--1421",
booktitle = "2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings",

}

TY - GEN

T1 - Distributed Stochastic Gradient Descent Using LDGM Codes

AU - Horii, Shunsuke

AU - Yoshida, Takahiro

AU - Kobayashi, Manabu

AU - Matsushima, Toshiyasu

PY - 2019/7

Y1 - 2019/7

N2 - We consider a distributed learning problem in which the computation is carried out on a system consisting of a master node and multiple worker nodes. In such systems, the existence of slow-running machines called stragglers will cause a significant decrease in performance. Recently, coding theoretic framework, which is named Gradient Coding (GC), for mitigating stragglers in distributed learning has been established by Tandon et al. Most studies on GC are aiming at recovering the gradient information completely assuming that the Gradient Descent (GD) algorithm is used as a learning algorithm. On the other hand, if the Stochastic Gradient Descent (SGD) algorithm is used, it is not necessary to completely recover the gradient information, and its unbiased estimator is sufficient for the learning. In this paper, we propose a distributed SGD scheme using Low Density Generator Matrix (LDGM) codes. In the proposed system, it may take longer time than existing GC methods to recover the gradient information completely, however, it enables the master node to obtain a high-quality unbiased estimator of the gradient at low computational cost and it leads to overall performance improvement.

AB - We consider a distributed learning problem in which the computation is carried out on a system consisting of a master node and multiple worker nodes. In such systems, the existence of slow-running machines called stragglers will cause a significant decrease in performance. Recently, coding theoretic framework, which is named Gradient Coding (GC), for mitigating stragglers in distributed learning has been established by Tandon et al. Most studies on GC are aiming at recovering the gradient information completely assuming that the Gradient Descent (GD) algorithm is used as a learning algorithm. On the other hand, if the Stochastic Gradient Descent (SGD) algorithm is used, it is not necessary to completely recover the gradient information, and its unbiased estimator is sufficient for the learning. In this paper, we propose a distributed SGD scheme using Low Density Generator Matrix (LDGM) codes. In the proposed system, it may take longer time than existing GC methods to recover the gradient information completely, however, it enables the master node to obtain a high-quality unbiased estimator of the gradient at low computational cost and it leads to overall performance improvement.

UR - http://www.scopus.com/inward/record.url?scp=85073152748&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85073152748&partnerID=8YFLogxK

U2 - 10.1109/ISIT.2019.8849580

DO - 10.1109/ISIT.2019.8849580

M3 - Conference contribution

AN - SCOPUS:85073152748

T3 - IEEE International Symposium on Information Theory - Proceedings

SP - 1417

EP - 1421

BT - 2019 IEEE International Symposium on Information Theory, ISIT 2019 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

ER -