Accurate, dense 3D reconstruction of moving and still objects from dynamic stereo sequences based on Temporal Modified-RANSAC and feature-cut

Naotomo Tatematsu, Jun Ohya

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    2 Citations (Scopus)

    Abstract

    This paper improves the authors' conventional method for reconstructing the 3D structure of moving and still objects that are tracked in the video and/or depth image sequences acquired by moving cameras and/or range finder. The authors proposed a Temporal Modified-RANSAC based method [1] that (1) can discriminate each moving object from the still background in color image and depth image sequences acquired by moving stereo cameras or moving range finder, (2) can compute the stereo cameras' egomotion, (3) can compute the motion of each moving object, and (4) can reconstruct the 3D structure of each moving object and the background. However, the TMR based method has the following two problems concerning the 3D reconstruction: lack of accuracy of segmenting into each object's region and sparse 3D reconstructed points in each object's region. To solve these problems of our conventional method, this paper proposes a new 3D segmentation method that utilizes Graph-cut, which is frequently used for segmentation tasks. First, the proposed method tracks feature points in the color and depth image sequences so that 3D optical flows of the feature points in every N frames are obtained. Then, TMR classifies all the obtained 3D optical flows into regions (3D flow set) for the background and each moving object; simultaneously, the rotation matrix and the translation vector for each 3D flow set are computed. Next, Graph-Cut using the energy function that consists of color probability, structure probability and a-priori probability is performed so that pixels in each frame are segmented into object regions and the background region. Finally, 3D point clouds are obtained from the segmentation result image and depth image, and then the point clouds are merged using the rotation and translation from the N-th frame prior to the current frame so that 3D models for the background and each moving object are constructed with dense 3D point data.

    Original languageEnglish
    Title of host publicationProceedings of SPIE - The International Society for Optical Engineering
    Volume8301
    DOIs
    Publication statusPublished - 2012
    EventIntelligent Robots and Computer Vision XXIX: Algorithms and Techniques - Burlingame, CA
    Duration: 2012 Jan 232012 Jan 24

    Other

    OtherIntelligent Robots and Computer Vision XXIX: Algorithms and Techniques
    CityBurlingame, CA
    Period12/1/2312/1/24

    Fingerprint

    RANSAC
    3D Reconstruction
    Moving Objects
    Range finders
    Optical flows
    Cameras
    Color
    Image Sequence
    range finders
    Graph Cuts
    Segmentation
    Camera
    Optical Flow
    cameras
    Point Cloud
    Feature Point
    Image segmentation
    color
    Pixels
    Rotation matrix

    Keywords

    • 3D-reconstruction
    • Detect multiple moving objects
    • Egomotion
    • Temporal Modified-RANSAC

    ASJC Scopus subject areas

    • Applied Mathematics
    • Computer Science Applications
    • Electrical and Electronic Engineering
    • Electronic, Optical and Magnetic Materials
    • Condensed Matter Physics

    Cite this

    Accurate, dense 3D reconstruction of moving and still objects from dynamic stereo sequences based on Temporal Modified-RANSAC and feature-cut. / Tatematsu, Naotomo; Ohya, Jun.

    Proceedings of SPIE - The International Society for Optical Engineering. Vol. 8301 2012. 830105.

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    Tatematsu, N & Ohya, J 2012, Accurate, dense 3D reconstruction of moving and still objects from dynamic stereo sequences based on Temporal Modified-RANSAC and feature-cut. in Proceedings of SPIE - The International Society for Optical Engineering. vol. 8301, 830105, Intelligent Robots and Computer Vision XXIX: Algorithms and Techniques, Burlingame, CA, 12/1/23. https://doi.org/10.1117/12.908037
    Tatematsu, Naotomo ; Ohya, Jun. / Accurate, dense 3D reconstruction of moving and still objects from dynamic stereo sequences based on Temporal Modified-RANSAC and feature-cut. Proceedings of SPIE - The International Society for Optical Engineering. Vol. 8301 2012.
    @inproceedings{d0d10cc52bf54c879a421f8e4d4c8ee7,
    title = "Accurate, dense 3D reconstruction of moving and still objects from dynamic stereo sequences based on Temporal Modified-RANSAC and feature-cut",
    abstract = "This paper improves the authors' conventional method for reconstructing the 3D structure of moving and still objects that are tracked in the video and/or depth image sequences acquired by moving cameras and/or range finder. The authors proposed a Temporal Modified-RANSAC based method [1] that (1) can discriminate each moving object from the still background in color image and depth image sequences acquired by moving stereo cameras or moving range finder, (2) can compute the stereo cameras' egomotion, (3) can compute the motion of each moving object, and (4) can reconstruct the 3D structure of each moving object and the background. However, the TMR based method has the following two problems concerning the 3D reconstruction: lack of accuracy of segmenting into each object's region and sparse 3D reconstructed points in each object's region. To solve these problems of our conventional method, this paper proposes a new 3D segmentation method that utilizes Graph-cut, which is frequently used for segmentation tasks. First, the proposed method tracks feature points in the color and depth image sequences so that 3D optical flows of the feature points in every N frames are obtained. Then, TMR classifies all the obtained 3D optical flows into regions (3D flow set) for the background and each moving object; simultaneously, the rotation matrix and the translation vector for each 3D flow set are computed. Next, Graph-Cut using the energy function that consists of color probability, structure probability and a-priori probability is performed so that pixels in each frame are segmented into object regions and the background region. Finally, 3D point clouds are obtained from the segmentation result image and depth image, and then the point clouds are merged using the rotation and translation from the N-th frame prior to the current frame so that 3D models for the background and each moving object are constructed with dense 3D point data.",
    keywords = "3D-reconstruction, Detect multiple moving objects, Egomotion, Temporal Modified-RANSAC",
    author = "Naotomo Tatematsu and Jun Ohya",
    year = "2012",
    doi = "10.1117/12.908037",
    language = "English",
    isbn = "9780819489487",
    volume = "8301",
    booktitle = "Proceedings of SPIE - The International Society for Optical Engineering",

    }

    TY - GEN

    T1 - Accurate, dense 3D reconstruction of moving and still objects from dynamic stereo sequences based on Temporal Modified-RANSAC and feature-cut

    AU - Tatematsu, Naotomo

    AU - Ohya, Jun

    PY - 2012

    Y1 - 2012

    N2 - This paper improves the authors' conventional method for reconstructing the 3D structure of moving and still objects that are tracked in the video and/or depth image sequences acquired by moving cameras and/or range finder. The authors proposed a Temporal Modified-RANSAC based method [1] that (1) can discriminate each moving object from the still background in color image and depth image sequences acquired by moving stereo cameras or moving range finder, (2) can compute the stereo cameras' egomotion, (3) can compute the motion of each moving object, and (4) can reconstruct the 3D structure of each moving object and the background. However, the TMR based method has the following two problems concerning the 3D reconstruction: lack of accuracy of segmenting into each object's region and sparse 3D reconstructed points in each object's region. To solve these problems of our conventional method, this paper proposes a new 3D segmentation method that utilizes Graph-cut, which is frequently used for segmentation tasks. First, the proposed method tracks feature points in the color and depth image sequences so that 3D optical flows of the feature points in every N frames are obtained. Then, TMR classifies all the obtained 3D optical flows into regions (3D flow set) for the background and each moving object; simultaneously, the rotation matrix and the translation vector for each 3D flow set are computed. Next, Graph-Cut using the energy function that consists of color probability, structure probability and a-priori probability is performed so that pixels in each frame are segmented into object regions and the background region. Finally, 3D point clouds are obtained from the segmentation result image and depth image, and then the point clouds are merged using the rotation and translation from the N-th frame prior to the current frame so that 3D models for the background and each moving object are constructed with dense 3D point data.

    AB - This paper improves the authors' conventional method for reconstructing the 3D structure of moving and still objects that are tracked in the video and/or depth image sequences acquired by moving cameras and/or range finder. The authors proposed a Temporal Modified-RANSAC based method [1] that (1) can discriminate each moving object from the still background in color image and depth image sequences acquired by moving stereo cameras or moving range finder, (2) can compute the stereo cameras' egomotion, (3) can compute the motion of each moving object, and (4) can reconstruct the 3D structure of each moving object and the background. However, the TMR based method has the following two problems concerning the 3D reconstruction: lack of accuracy of segmenting into each object's region and sparse 3D reconstructed points in each object's region. To solve these problems of our conventional method, this paper proposes a new 3D segmentation method that utilizes Graph-cut, which is frequently used for segmentation tasks. First, the proposed method tracks feature points in the color and depth image sequences so that 3D optical flows of the feature points in every N frames are obtained. Then, TMR classifies all the obtained 3D optical flows into regions (3D flow set) for the background and each moving object; simultaneously, the rotation matrix and the translation vector for each 3D flow set are computed. Next, Graph-Cut using the energy function that consists of color probability, structure probability and a-priori probability is performed so that pixels in each frame are segmented into object regions and the background region. Finally, 3D point clouds are obtained from the segmentation result image and depth image, and then the point clouds are merged using the rotation and translation from the N-th frame prior to the current frame so that 3D models for the background and each moving object are constructed with dense 3D point data.

    KW - 3D-reconstruction

    KW - Detect multiple moving objects

    KW - Egomotion

    KW - Temporal Modified-RANSAC

    UR - http://www.scopus.com/inward/record.url?scp=84857001761&partnerID=8YFLogxK

    UR - http://www.scopus.com/inward/citedby.url?scp=84857001761&partnerID=8YFLogxK

    U2 - 10.1117/12.908037

    DO - 10.1117/12.908037

    M3 - Conference contribution

    AN - SCOPUS:84857001761

    SN - 9780819489487

    VL - 8301

    BT - Proceedings of SPIE - The International Society for Optical Engineering

    ER -