MEC: Network optimized multi-stage erasure coding for scalable storage systems

Hiroaki Akutsu, Takahiro Yamamoto, Kazunori Ueda, Hideo Saito

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Abstract

    In scalable storage systems, there are two kinds of methods for data redundancy: mirroring and parity. Each has its pros and cons. Mirroring creates a large amount of redundancy data, resulting in less usable space. Write performance degrades proportionally to the redundancy level due to an increase in communication. Parity-based methods partition data into multiple pieces, add parity information, and distribute the pieces of data and parity information. Parity-based methods are not often used with memory class media that are faster than the network, because distributing the data across servers results in low read performance. This research aims to establish an efficient data protection method that can be applied to fast, memory class media. We propose a new parity-based method called Multi-stage Erasure Coding (MEC), which creates two different erasure codes: one at the data transmission source server, and the other at the destination server. We show that our method reduces the space required to achieve redundancy while achieving high performance by making the amount of write communication independent of the redundancy level. We built a prototype program using MEC on a commodity cluster server. We show that compared with conventional parity-based methods with redundancy level 2, read I/O throughput is over one order of magnitude higher thanks to local reads and that write I/O throughput is almost the same due to network bottleneck.

    Original languageEnglish
    Title of host publicationProceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017
    PublisherIEEE Computer Society
    Pages292-300
    Number of pages9
    ISBN (Electronic)9781509056514
    DOIs
    Publication statusPublished - 2017 May 5
    Event22nd IEEE Pacific Rim International Symposium on Dependable Computing, PRDC 2017 - Christchurch, New Zealand
    Duration: 2017 Jan 222017 Jan 25

    Other

    Other22nd IEEE Pacific Rim International Symposium on Dependable Computing, PRDC 2017
    CountryNew Zealand
    CityChristchurch
    Period17/1/2217/1/25

    Keywords

    • Erasure coding
    • High performance computing
    • Rebuild
    • Scalablility
    • Storage systems

    ASJC Scopus subject areas

    • Computational Theory and Mathematics
    • Computer Science Applications
    • Hardware and Architecture
    • Software

    Fingerprint Dive into the research topics of 'MEC: Network optimized multi-stage erasure coding for scalable storage systems'. Together they form a unique fingerprint.

  • Cite this

    Akutsu, H., Yamamoto, T., Ueda, K., & Saito, H. (2017). MEC: Network optimized multi-stage erasure coding for scalable storage systems. In Proceedings - 2017 IEEE 22nd Pacific Rim International Symposium on Dependable Computing, PRDC 2017 (pp. 292-300). [7920634] IEEE Computer Society. https://doi.org/10.1109/PRDC.2017.54