Phasebook and friends

Leveraging discrete representations for source separation

Jonathan Le Roux, Gordon Wichern, Shinji Watanabe, Andy Sarroff, John R. Hershey

Research output: Contribution to journalArticle

Abstract

Speech enhancement and source separation systems based on deep learning have recently reached unprecedented levels of quality, to the point that performance is reaching a new ceiling. Most systems rely on estimating the magnitude of a target source by estimating a real-valued mask to be applied to a time-frequency representation of the mixture signal. A limiting factor in such approaches is a lack of phase estimation: the phase of the mixture is most often used when reconstructing the estimated time-domain signal. Here, we propose 'magbook,' 'phasebook,' and 'combook,' three new types of layers based on discrete representations that can be used to estimate complex time-frequency masks. Magbook layers extend classical sigmoidal units and a recently introduced convex softmax activation for mask-based magnitude estimation. Phasebook layers use a similar structure to give an estimate of the phase mask without suffering from phase wrapping issues. Combook layers are an alternative to the magbook-phasebook combination that directly estimate complex masks. We present various training and inference schemes involving these representations, and explain in particular how to include them in an end-to-end learning framework. We also present an oracle study to assess upper bounds on performance for various types of masks using discrete phase representations. We evaluate the proposed methods on the wsj0-2mix dataset, a well-studied corpus for single-channel speaker-independent speaker separation, matching the performance of state-of-the-art mask-based approaches without requiring additional phase reconstruction steps.

Original languageEnglish
Article number8664086
Pages (from-to)370-382
Number of pages13
JournalIEEE Journal on Selected Topics in Signal Processing
Volume13
Issue number2
DOIs
Publication statusPublished - 2019 May 1
Externally publishedYes

Fingerprint

Source separation
Masks
Speech enhancement
Ceilings
Chemical activation

Keywords

  • deep clustering
  • deep learning
  • discrete representation
  • mask inference
  • phase
  • quantization
  • Source separation

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering

Cite this

Phasebook and friends : Leveraging discrete representations for source separation. / Le Roux, Jonathan; Wichern, Gordon; Watanabe, Shinji; Sarroff, Andy; Hershey, John R.

In: IEEE Journal on Selected Topics in Signal Processing, Vol. 13, No. 2, 8664086, 01.05.2019, p. 370-382.

Research output: Contribution to journalArticle

Le Roux, Jonathan ; Wichern, Gordon ; Watanabe, Shinji ; Sarroff, Andy ; Hershey, John R. / Phasebook and friends : Leveraging discrete representations for source separation. In: IEEE Journal on Selected Topics in Signal Processing. 2019 ; Vol. 13, No. 2. pp. 370-382.
@article{cb62f9022e15465987e3a2b949b7d0dc,
title = "Phasebook and friends: Leveraging discrete representations for source separation",
abstract = "Speech enhancement and source separation systems based on deep learning have recently reached unprecedented levels of quality, to the point that performance is reaching a new ceiling. Most systems rely on estimating the magnitude of a target source by estimating a real-valued mask to be applied to a time-frequency representation of the mixture signal. A limiting factor in such approaches is a lack of phase estimation: the phase of the mixture is most often used when reconstructing the estimated time-domain signal. Here, we propose 'magbook,' 'phasebook,' and 'combook,' three new types of layers based on discrete representations that can be used to estimate complex time-frequency masks. Magbook layers extend classical sigmoidal units and a recently introduced convex softmax activation for mask-based magnitude estimation. Phasebook layers use a similar structure to give an estimate of the phase mask without suffering from phase wrapping issues. Combook layers are an alternative to the magbook-phasebook combination that directly estimate complex masks. We present various training and inference schemes involving these representations, and explain in particular how to include them in an end-to-end learning framework. We also present an oracle study to assess upper bounds on performance for various types of masks using discrete phase representations. We evaluate the proposed methods on the wsj0-2mix dataset, a well-studied corpus for single-channel speaker-independent speaker separation, matching the performance of state-of-the-art mask-based approaches without requiring additional phase reconstruction steps.",
keywords = "deep clustering, deep learning, discrete representation, mask inference, phase, quantization, Source separation",
author = "{Le Roux}, Jonathan and Gordon Wichern and Shinji Watanabe and Andy Sarroff and Hershey, {John R.}",
year = "2019",
month = "5",
day = "1",
doi = "10.1109/JSTSP.2019.2904183",
language = "English",
volume = "13",
pages = "370--382",
journal = "IEEE Journal on Selected Topics in Signal Processing",
issn = "1932-4553",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "2",

}

TY - JOUR

T1 - Phasebook and friends

T2 - Leveraging discrete representations for source separation

AU - Le Roux, Jonathan

AU - Wichern, Gordon

AU - Watanabe, Shinji

AU - Sarroff, Andy

AU - Hershey, John R.

PY - 2019/5/1

Y1 - 2019/5/1

N2 - Speech enhancement and source separation systems based on deep learning have recently reached unprecedented levels of quality, to the point that performance is reaching a new ceiling. Most systems rely on estimating the magnitude of a target source by estimating a real-valued mask to be applied to a time-frequency representation of the mixture signal. A limiting factor in such approaches is a lack of phase estimation: the phase of the mixture is most often used when reconstructing the estimated time-domain signal. Here, we propose 'magbook,' 'phasebook,' and 'combook,' three new types of layers based on discrete representations that can be used to estimate complex time-frequency masks. Magbook layers extend classical sigmoidal units and a recently introduced convex softmax activation for mask-based magnitude estimation. Phasebook layers use a similar structure to give an estimate of the phase mask without suffering from phase wrapping issues. Combook layers are an alternative to the magbook-phasebook combination that directly estimate complex masks. We present various training and inference schemes involving these representations, and explain in particular how to include them in an end-to-end learning framework. We also present an oracle study to assess upper bounds on performance for various types of masks using discrete phase representations. We evaluate the proposed methods on the wsj0-2mix dataset, a well-studied corpus for single-channel speaker-independent speaker separation, matching the performance of state-of-the-art mask-based approaches without requiring additional phase reconstruction steps.

AB - Speech enhancement and source separation systems based on deep learning have recently reached unprecedented levels of quality, to the point that performance is reaching a new ceiling. Most systems rely on estimating the magnitude of a target source by estimating a real-valued mask to be applied to a time-frequency representation of the mixture signal. A limiting factor in such approaches is a lack of phase estimation: the phase of the mixture is most often used when reconstructing the estimated time-domain signal. Here, we propose 'magbook,' 'phasebook,' and 'combook,' three new types of layers based on discrete representations that can be used to estimate complex time-frequency masks. Magbook layers extend classical sigmoidal units and a recently introduced convex softmax activation for mask-based magnitude estimation. Phasebook layers use a similar structure to give an estimate of the phase mask without suffering from phase wrapping issues. Combook layers are an alternative to the magbook-phasebook combination that directly estimate complex masks. We present various training and inference schemes involving these representations, and explain in particular how to include them in an end-to-end learning framework. We also present an oracle study to assess upper bounds on performance for various types of masks using discrete phase representations. We evaluate the proposed methods on the wsj0-2mix dataset, a well-studied corpus for single-channel speaker-independent speaker separation, matching the performance of state-of-the-art mask-based approaches without requiring additional phase reconstruction steps.

KW - deep clustering

KW - deep learning

KW - discrete representation

KW - mask inference

KW - phase

KW - quantization

KW - Source separation

UR - http://www.scopus.com/inward/record.url?scp=85065923984&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85065923984&partnerID=8YFLogxK

U2 - 10.1109/JSTSP.2019.2904183

DO - 10.1109/JSTSP.2019.2904183

M3 - Article

VL - 13

SP - 370

EP - 382

JO - IEEE Journal on Selected Topics in Signal Processing

JF - IEEE Journal on Selected Topics in Signal Processing

SN - 1932-4553

IS - 2

M1 - 8664086

ER -