Open code coverage framework

A framework for consistent, flexible and complete measurement of test coverage supporting multiple programming languages

Kazunori Sakamoto, Fuyuki Ishikawa, Hironori Washizaki, Yoshiaki Fukazawa

    Research output: Contribution to journalArticle

    1 Citation (Scopus)

    Abstract

    Test coverage is an important indicator of whether software has been sufficiently tested. However, there are several problems with the existing measurement tools for test coverage, such as their cost of development and maintenance, inconsistency, and inflexibility in measurement. We propose a consistent and flexible measurement framework for test coverage that we call the Open Code Coverage Framework (OCCF). It supports multiple programming languages by extracting the commonalities from multiple programming languages using an abstract syntax tree to help in the development of the measurement tools for the test coverage of new programming languages. OCCF allows users to add programming language support independently of the test-coverage-criteria and also to add test-coverage-criteria support independently of programming languages in order to take consistent measurements in each programming language. Moreover, OCCF provides two methods for changin the measurement range and elements using XPath and adding user code in order to make more flexible measurements. We implemented a sample tool for C, Java, and Python using OCCF. OCCF can measure four test-coveragecriteria. We also confirmed that OCCF can support C#, Ruby, JavaScript, and Lua. Moreover, we reduced the lines of code (LOCs) required to implement measurement tools for test coverage by approximately 90% and the time to implement a new test-coverage-criterion by over 80% in an experiment that compared OCCF with the conventional non-framework-based tools.

    Original languageEnglish
    Pages (from-to)2418-2430
    Number of pages13
    JournalIEICE Transactions on Information and Systems
    VolumeE94-D
    Issue number12
    DOIs
    Publication statusPublished - 2011 Dec

    Fingerprint

    Computer programming languages
    Ruby
    Costs
    Experiments

    Keywords

    • Code coverage
    • Framework
    • Metrics
    • Software testing
    • Test coverage

    ASJC Scopus subject areas

    • Electrical and Electronic Engineering
    • Software
    • Artificial Intelligence
    • Hardware and Architecture
    • Computer Vision and Pattern Recognition

    Cite this

    @article{156d1a7576624b97a995882e8ce8ce62,
    title = "Open code coverage framework: A framework for consistent, flexible and complete measurement of test coverage supporting multiple programming languages",
    abstract = "Test coverage is an important indicator of whether software has been sufficiently tested. However, there are several problems with the existing measurement tools for test coverage, such as their cost of development and maintenance, inconsistency, and inflexibility in measurement. We propose a consistent and flexible measurement framework for test coverage that we call the Open Code Coverage Framework (OCCF). It supports multiple programming languages by extracting the commonalities from multiple programming languages using an abstract syntax tree to help in the development of the measurement tools for the test coverage of new programming languages. OCCF allows users to add programming language support independently of the test-coverage-criteria and also to add test-coverage-criteria support independently of programming languages in order to take consistent measurements in each programming language. Moreover, OCCF provides two methods for changin the measurement range and elements using XPath and adding user code in order to make more flexible measurements. We implemented a sample tool for C, Java, and Python using OCCF. OCCF can measure four test-coveragecriteria. We also confirmed that OCCF can support C#, Ruby, JavaScript, and Lua. Moreover, we reduced the lines of code (LOCs) required to implement measurement tools for test coverage by approximately 90{\%} and the time to implement a new test-coverage-criterion by over 80{\%} in an experiment that compared OCCF with the conventional non-framework-based tools.",
    keywords = "Code coverage, Framework, Metrics, Software testing, Test coverage",
    author = "Kazunori Sakamoto and Fuyuki Ishikawa and Hironori Washizaki and Yoshiaki Fukazawa",
    year = "2011",
    month = "12",
    doi = "10.1587/transinf.E94.D.2418",
    language = "English",
    volume = "E94-D",
    pages = "2418--2430",
    journal = "IEICE Transactions on Information and Systems",
    issn = "0916-8532",
    publisher = "Maruzen Co., Ltd/Maruzen Kabushikikaisha",
    number = "12",

    }

    TY - JOUR

    T1 - Open code coverage framework

    T2 - A framework for consistent, flexible and complete measurement of test coverage supporting multiple programming languages

    AU - Sakamoto, Kazunori

    AU - Ishikawa, Fuyuki

    AU - Washizaki, Hironori

    AU - Fukazawa, Yoshiaki

    PY - 2011/12

    Y1 - 2011/12

    N2 - Test coverage is an important indicator of whether software has been sufficiently tested. However, there are several problems with the existing measurement tools for test coverage, such as their cost of development and maintenance, inconsistency, and inflexibility in measurement. We propose a consistent and flexible measurement framework for test coverage that we call the Open Code Coverage Framework (OCCF). It supports multiple programming languages by extracting the commonalities from multiple programming languages using an abstract syntax tree to help in the development of the measurement tools for the test coverage of new programming languages. OCCF allows users to add programming language support independently of the test-coverage-criteria and also to add test-coverage-criteria support independently of programming languages in order to take consistent measurements in each programming language. Moreover, OCCF provides two methods for changin the measurement range and elements using XPath and adding user code in order to make more flexible measurements. We implemented a sample tool for C, Java, and Python using OCCF. OCCF can measure four test-coveragecriteria. We also confirmed that OCCF can support C#, Ruby, JavaScript, and Lua. Moreover, we reduced the lines of code (LOCs) required to implement measurement tools for test coverage by approximately 90% and the time to implement a new test-coverage-criterion by over 80% in an experiment that compared OCCF with the conventional non-framework-based tools.

    AB - Test coverage is an important indicator of whether software has been sufficiently tested. However, there are several problems with the existing measurement tools for test coverage, such as their cost of development and maintenance, inconsistency, and inflexibility in measurement. We propose a consistent and flexible measurement framework for test coverage that we call the Open Code Coverage Framework (OCCF). It supports multiple programming languages by extracting the commonalities from multiple programming languages using an abstract syntax tree to help in the development of the measurement tools for the test coverage of new programming languages. OCCF allows users to add programming language support independently of the test-coverage-criteria and also to add test-coverage-criteria support independently of programming languages in order to take consistent measurements in each programming language. Moreover, OCCF provides two methods for changin the measurement range and elements using XPath and adding user code in order to make more flexible measurements. We implemented a sample tool for C, Java, and Python using OCCF. OCCF can measure four test-coveragecriteria. We also confirmed that OCCF can support C#, Ruby, JavaScript, and Lua. Moreover, we reduced the lines of code (LOCs) required to implement measurement tools for test coverage by approximately 90% and the time to implement a new test-coverage-criterion by over 80% in an experiment that compared OCCF with the conventional non-framework-based tools.

    KW - Code coverage

    KW - Framework

    KW - Metrics

    KW - Software testing

    KW - Test coverage

    UR - http://www.scopus.com/inward/record.url?scp=82655165273&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=82655165273&partnerID=8YFLogxK

    U2 - 10.1587/transinf.E94.D.2418

    DO - 10.1587/transinf.E94.D.2418

    M3 - Article

    VL - E94-D

    SP - 2418

    EP - 2430

    JO - IEICE Transactions on Information and Systems

    JF - IEICE Transactions on Information and Systems

    SN - 0916-8532

    IS - 12

    ER -