MEC: Network optimized multi-stage erasure coding for scalable storage systems

Hiroaki Akutsu, Takahiro Yamamoto, Kazunori Ueda, Hideo Saito

    研究成果: Conference contribution

    抄録

    In scalable storage systems, there are two kinds of methods for data redundancy: mirroring and parity. Each has its pros and cons. Mirroring creates a large amount of redundancy data, resulting in less usable space. Write performance degrades proportionally to the redundancy level due to an increase in communication. Parity-based methods partition data into multiple pieces, add parity information, and distribute the pieces of data and parity information. Parity-based methods are not often used with memory class media that are faster than the network, because distributing the data across servers results in low read performance. This research aims to establish an efficient data protection method that can be applied to fast, memory class media. We propose a new parity-based method called Multi-stage Erasure Coding (MEC), which creates two different erasure codes: one at the data transmission source server, and the other at the destination server. We show that our method reduces the space required to achieve redundancy while achieving high performance by making the amount of write communication independent of the redundancy level. We built a prototype program using MEC on a commodity cluster server. We show that compared with conventional parity-based methods with redundancy level 2, read I/O throughput is over one order of magnitude higher thanks to local reads and that write I/O throughput is almost the same due to network bottleneck.

    元の言語English
    ホスト出版物のタイトルProceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017
    出版者IEEE Computer Society
    ページ292-300
    ページ数9
    ISBN(電子版)9781509056514
    DOI
    出版物ステータスPublished - 2017 5 5
    イベント22nd IEEE Pacific Rim International Symposium on Dependable Computing, PRDC 2017 - Christchurch, New Zealand
    継続期間: 2017 1 222017 1 25

    Other

    Other22nd IEEE Pacific Rim International Symposium on Dependable Computing, PRDC 2017
    New Zealand
    Christchurch
    期間17/1/2217/1/25

    Fingerprint

    Network coding
    Redundancy
    Servers
    Throughput
    Data storage equipment
    Data privacy
    Communication
    Data communication systems

    ASJC Scopus subject areas

    • Computational Theory and Mathematics
    • Computer Science Applications
    • Hardware and Architecture
    • Software

    これを引用

    Akutsu, H., Yamamoto, T., Ueda, K., & Saito, H. (2017). MEC: Network optimized multi-stage erasure coding for scalable storage systems. : Proceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017 (pp. 292-300). [7920634] IEEE Computer Society. https://doi.org/10.1109/PRDC.2017.54

    MEC : Network optimized multi-stage erasure coding for scalable storage systems. / Akutsu, Hiroaki; Yamamoto, Takahiro; Ueda, Kazunori; Saito, Hideo.

    Proceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017. IEEE Computer Society, 2017. p. 292-300 7920634.

    研究成果: Conference contribution

    Akutsu, H, Yamamoto, T, Ueda, K & Saito, H 2017, MEC: Network optimized multi-stage erasure coding for scalable storage systems. : Proceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017., 7920634, IEEE Computer Society, pp. 292-300, 22nd IEEE Pacific Rim International Symposium on Dependable Computing, PRDC 2017, Christchurch, New Zealand, 17/1/22. https://doi.org/10.1109/PRDC.2017.54
    Akutsu H, Yamamoto T, Ueda K, Saito H. MEC: Network optimized multi-stage erasure coding for scalable storage systems. : Proceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017. IEEE Computer Society. 2017. p. 292-300. 7920634 https://doi.org/10.1109/PRDC.2017.54
    Akutsu, Hiroaki ; Yamamoto, Takahiro ; Ueda, Kazunori ; Saito, Hideo. / MEC : Network optimized multi-stage erasure coding for scalable storage systems. Proceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017. IEEE Computer Society, 2017. pp. 292-300
    @inproceedings{553c24bef2d943e0bb76a30b0b464bc9,
    title = "MEC: Network optimized multi-stage erasure coding for scalable storage systems",
    abstract = "In scalable storage systems, there are two kinds of methods for data redundancy: mirroring and parity. Each has its pros and cons. Mirroring creates a large amount of redundancy data, resulting in less usable space. Write performance degrades proportionally to the redundancy level due to an increase in communication. Parity-based methods partition data into multiple pieces, add parity information, and distribute the pieces of data and parity information. Parity-based methods are not often used with memory class media that are faster than the network, because distributing the data across servers results in low read performance. This research aims to establish an efficient data protection method that can be applied to fast, memory class media. We propose a new parity-based method called Multi-stage Erasure Coding (MEC), which creates two different erasure codes: one at the data transmission source server, and the other at the destination server. We show that our method reduces the space required to achieve redundancy while achieving high performance by making the amount of write communication independent of the redundancy level. We built a prototype program using MEC on a commodity cluster server. We show that compared with conventional parity-based methods with redundancy level 2, read I/O throughput is over one order of magnitude higher thanks to local reads and that write I/O throughput is almost the same due to network bottleneck.",
    keywords = "Erasure coding, High performance computing, Rebuild, Scalablility, Storage systems",
    author = "Hiroaki Akutsu and Takahiro Yamamoto and Kazunori Ueda and Hideo Saito",
    year = "2017",
    month = "5",
    day = "5",
    doi = "10.1109/PRDC.2017.54",
    language = "English",
    pages = "292--300",
    booktitle = "Proceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017",
    publisher = "IEEE Computer Society",
    address = "United States",

    }

    TY - GEN

    T1 - MEC

    T2 - Network optimized multi-stage erasure coding for scalable storage systems

    AU - Akutsu, Hiroaki

    AU - Yamamoto, Takahiro

    AU - Ueda, Kazunori

    AU - Saito, Hideo

    PY - 2017/5/5

    Y1 - 2017/5/5

    N2 - In scalable storage systems, there are two kinds of methods for data redundancy: mirroring and parity. Each has its pros and cons. Mirroring creates a large amount of redundancy data, resulting in less usable space. Write performance degrades proportionally to the redundancy level due to an increase in communication. Parity-based methods partition data into multiple pieces, add parity information, and distribute the pieces of data and parity information. Parity-based methods are not often used with memory class media that are faster than the network, because distributing the data across servers results in low read performance. This research aims to establish an efficient data protection method that can be applied to fast, memory class media. We propose a new parity-based method called Multi-stage Erasure Coding (MEC), which creates two different erasure codes: one at the data transmission source server, and the other at the destination server. We show that our method reduces the space required to achieve redundancy while achieving high performance by making the amount of write communication independent of the redundancy level. We built a prototype program using MEC on a commodity cluster server. We show that compared with conventional parity-based methods with redundancy level 2, read I/O throughput is over one order of magnitude higher thanks to local reads and that write I/O throughput is almost the same due to network bottleneck.

    AB - In scalable storage systems, there are two kinds of methods for data redundancy: mirroring and parity. Each has its pros and cons. Mirroring creates a large amount of redundancy data, resulting in less usable space. Write performance degrades proportionally to the redundancy level due to an increase in communication. Parity-based methods partition data into multiple pieces, add parity information, and distribute the pieces of data and parity information. Parity-based methods are not often used with memory class media that are faster than the network, because distributing the data across servers results in low read performance. This research aims to establish an efficient data protection method that can be applied to fast, memory class media. We propose a new parity-based method called Multi-stage Erasure Coding (MEC), which creates two different erasure codes: one at the data transmission source server, and the other at the destination server. We show that our method reduces the space required to achieve redundancy while achieving high performance by making the amount of write communication independent of the redundancy level. We built a prototype program using MEC on a commodity cluster server. We show that compared with conventional parity-based methods with redundancy level 2, read I/O throughput is over one order of magnitude higher thanks to local reads and that write I/O throughput is almost the same due to network bottleneck.

    KW - Erasure coding

    KW - High performance computing

    KW - Rebuild

    KW - Scalablility

    KW - Storage systems

    UR - http://www.scopus.com/inward/record.url?scp=85019636571&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=85019636571&partnerID=8YFLogxK

    U2 - 10.1109/PRDC.2017.54

    DO - 10.1109/PRDC.2017.54

    M3 - Conference contribution

    AN - SCOPUS:85019636571

    SP - 292

    EP - 300

    BT - Proceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017

    PB - IEEE Computer Society

    ER -