Sparse Bayesian Hierarchical Mixture of Experts and Variational Inference

研究成果: Conference contribution

1 引用 (Scopus)

抄録

The hierarchical mixture of experts (HME) is a tree-structured probabilistic model for regression and classification. The HME has a considerable expression capability, however, the estimation of the parameters tends to overfit due to the complexity of the model. To avoid this problem, regularization techniques are widely used. In particular, it is known that a sparse solution can be obtained by L1 regularization. From a Bayesian point of view, regularization techniques are equivalent to assume that the parameters follow prior distributions and find the maximum a posteriori probability estimator. It is known that L1 regularization is equivalent to assuming Laplace distributions as prior distributions. However, it is difficult to compute the posterior distribution if Laplace distributions are assumed. In this paper, we assume that the parameters of the HME follow hierarchical prior distributions which are equivalent to Laplace distribution to promote sparse solutions. We propose a Bayesian estimation algorithm based on the variational method. Finally, the proposed algorithm is evaluated by computer simulations.

元の言語English
ホスト出版物のタイトルProceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018
出版者Institute of Electrical and Electronics Engineers Inc.
ページ60-64
ページ数5
ISBN(電子版)9784885523182
DOI
出版物ステータスPublished - 2019 3 8
イベント15th International Symposium on Information Theory and Its Applications, ISITA 2018 - Singapore, Singapore
継続期間: 2018 10 282018 10 31

出版物シリーズ

名前Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018

Conference

Conference15th International Symposium on Information Theory and Its Applications, ISITA 2018
Singapore
Singapore
期間18/10/2818/10/31

Fingerprint

Computer simulation
Statistical Models

ASJC Scopus subject areas

  • Computer Science Applications
  • Information Systems

これを引用

Iikubo, Y., Horii, S., & Matsushima, T. (2019). Sparse Bayesian Hierarchical Mixture of Experts and Variational Inference. : Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018 (pp. 60-64). [8664333] (Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.23919/ISITA.2018.8664333

Sparse Bayesian Hierarchical Mixture of Experts and Variational Inference. / Iikubo, Yuji; Horii, Shunsuke; Matsushima, Toshiyasu.

Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 60-64 8664333 (Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018).

研究成果: Conference contribution

Iikubo, Y, Horii, S & Matsushima, T 2019, Sparse Bayesian Hierarchical Mixture of Experts and Variational Inference. : Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018., 8664333, Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018, Institute of Electrical and Electronics Engineers Inc., pp. 60-64, 15th International Symposium on Information Theory and Its Applications, ISITA 2018, Singapore, Singapore, 18/10/28. https://doi.org/10.23919/ISITA.2018.8664333
Iikubo Y, Horii S, Matsushima T. Sparse Bayesian Hierarchical Mixture of Experts and Variational Inference. : Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 60-64. 8664333. (Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018). https://doi.org/10.23919/ISITA.2018.8664333
Iikubo, Yuji ; Horii, Shunsuke ; Matsushima, Toshiyasu. / Sparse Bayesian Hierarchical Mixture of Experts and Variational Inference. Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 60-64 (Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018).
@inproceedings{817885d7a34540268a305dd378a6bd32,
title = "Sparse Bayesian Hierarchical Mixture of Experts and Variational Inference",
abstract = "The hierarchical mixture of experts (HME) is a tree-structured probabilistic model for regression and classification. The HME has a considerable expression capability, however, the estimation of the parameters tends to overfit due to the complexity of the model. To avoid this problem, regularization techniques are widely used. In particular, it is known that a sparse solution can be obtained by L1 regularization. From a Bayesian point of view, regularization techniques are equivalent to assume that the parameters follow prior distributions and find the maximum a posteriori probability estimator. It is known that L1 regularization is equivalent to assuming Laplace distributions as prior distributions. However, it is difficult to compute the posterior distribution if Laplace distributions are assumed. In this paper, we assume that the parameters of the HME follow hierarchical prior distributions which are equivalent to Laplace distribution to promote sparse solutions. We propose a Bayesian estimation algorithm based on the variational method. Finally, the proposed algorithm is evaluated by computer simulations.",
author = "Yuji Iikubo and Shunsuke Horii and Toshiyasu Matsushima",
year = "2019",
month = "3",
day = "8",
doi = "10.23919/ISITA.2018.8664333",
language = "English",
series = "Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "60--64",
booktitle = "Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018",

}

TY - GEN

T1 - Sparse Bayesian Hierarchical Mixture of Experts and Variational Inference

AU - Iikubo, Yuji

AU - Horii, Shunsuke

AU - Matsushima, Toshiyasu

PY - 2019/3/8

Y1 - 2019/3/8

N2 - The hierarchical mixture of experts (HME) is a tree-structured probabilistic model for regression and classification. The HME has a considerable expression capability, however, the estimation of the parameters tends to overfit due to the complexity of the model. To avoid this problem, regularization techniques are widely used. In particular, it is known that a sparse solution can be obtained by L1 regularization. From a Bayesian point of view, regularization techniques are equivalent to assume that the parameters follow prior distributions and find the maximum a posteriori probability estimator. It is known that L1 regularization is equivalent to assuming Laplace distributions as prior distributions. However, it is difficult to compute the posterior distribution if Laplace distributions are assumed. In this paper, we assume that the parameters of the HME follow hierarchical prior distributions which are equivalent to Laplace distribution to promote sparse solutions. We propose a Bayesian estimation algorithm based on the variational method. Finally, the proposed algorithm is evaluated by computer simulations.

AB - The hierarchical mixture of experts (HME) is a tree-structured probabilistic model for regression and classification. The HME has a considerable expression capability, however, the estimation of the parameters tends to overfit due to the complexity of the model. To avoid this problem, regularization techniques are widely used. In particular, it is known that a sparse solution can be obtained by L1 regularization. From a Bayesian point of view, regularization techniques are equivalent to assume that the parameters follow prior distributions and find the maximum a posteriori probability estimator. It is known that L1 regularization is equivalent to assuming Laplace distributions as prior distributions. However, it is difficult to compute the posterior distribution if Laplace distributions are assumed. In this paper, we assume that the parameters of the HME follow hierarchical prior distributions which are equivalent to Laplace distribution to promote sparse solutions. We propose a Bayesian estimation algorithm based on the variational method. Finally, the proposed algorithm is evaluated by computer simulations.

UR - http://www.scopus.com/inward/record.url?scp=85063870175&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063870175&partnerID=8YFLogxK

U2 - 10.23919/ISITA.2018.8664333

DO - 10.23919/ISITA.2018.8664333

M3 - Conference contribution

AN - SCOPUS:85063870175

T3 - Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018

SP - 60

EP - 64

BT - Proceedings of 2018 International Symposium on Information Theory and Its Applications, ISITA 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -