Multiple description coding and recovery of free viewpoint video for wireless multi-path streaming

Zhi Liu, Gene Cheung, Jacob Chakareski, Yusheng Ji

Research output: Contribution to journalArticle

34 Citations (Scopus)

Abstract

By transmitting texture and depth videos captured from two nearby camera viewpoints, a client can synthesize via depth-image-based rendering (DIBR) any freely chosen intermediate virtual view of the 3D scene, enhancing the user's perception of depth. During wireless network transmission, burst packet losses can corrupt the transmitted texture and depth videos and degrade the synthesized view quality at the client. In this paper, we propose a multiple description coding system for multi-path transmission of free-viewpoint video, with joint interview and temporal description recovery capability. In particular, we encode separately the even frames of the left view and the odd frames of the right view, and transmit them as one description on one path. The second description comprises the remaining frames in the two views and is transmitted over a second path. If the receiver receives only one description due to burst loss in the other path, the missing frames in the other description are partially reconstructed using our frame recovery procedure. First, we construct two recovery candidates for each lost pixel in a frame. The first candidate is generated via temporal super-resolution from its predecessor and successor frames in the same view. The second candidate is generated via DIBR fromthe received frame of the same time instance in the other view. Next, we select the best pixel candidates one patch at a time, where an image patch corresponds to a neighborhood of pixels with similar depth values in the 3D scene. Near-optimal source and channel coding rates for each description are selected using a branch-and-bound method, for given transmission bandwidth on each path. Experimental results show that our systemcan outperform a traditional single-description/single-path transmission scheme by up to 5.5 dB in Peak Signal-to-Noise Ratio (PSNR) of the synthesized intermediate view at the client.

Original languageEnglish
Article number6832458
Pages (from-to)151-164
Number of pages14
JournalIEEE Journal on Selected Topics in Signal Processing
Volume9
Issue number1
DOIs
Publication statusPublished - 2015 Feb 1
Externally publishedYes

Fingerprint

Recovery
Pixels
Textures
Branch and bound method
Channel coding
Packet loss
Wireless networks
Signal to noise ratio
Cameras
Bandwidth

Keywords

  • Depth-image-based rendering
  • Free-viewpoints video
  • Multiple description coding
  • Wireless multi-path streaming

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Signal Processing

Cite this

Multiple description coding and recovery of free viewpoint video for wireless multi-path streaming. / Liu, Zhi; Cheung, Gene; Chakareski, Jacob; Ji, Yusheng.

In: IEEE Journal on Selected Topics in Signal Processing, Vol. 9, No. 1, 6832458, 01.02.2015, p. 151-164.

Research output: Contribution to journalArticle

Liu, Zhi ; Cheung, Gene ; Chakareski, Jacob ; Ji, Yusheng. / Multiple description coding and recovery of free viewpoint video for wireless multi-path streaming. In: IEEE Journal on Selected Topics in Signal Processing. 2015 ; Vol. 9, No. 1. pp. 151-164.
@article{5ba406d4d442455cb51eea23be582435,
title = "Multiple description coding and recovery of free viewpoint video for wireless multi-path streaming",
abstract = "By transmitting texture and depth videos captured from two nearby camera viewpoints, a client can synthesize via depth-image-based rendering (DIBR) any freely chosen intermediate virtual view of the 3D scene, enhancing the user's perception of depth. During wireless network transmission, burst packet losses can corrupt the transmitted texture and depth videos and degrade the synthesized view quality at the client. In this paper, we propose a multiple description coding system for multi-path transmission of free-viewpoint video, with joint interview and temporal description recovery capability. In particular, we encode separately the even frames of the left view and the odd frames of the right view, and transmit them as one description on one path. The second description comprises the remaining frames in the two views and is transmitted over a second path. If the receiver receives only one description due to burst loss in the other path, the missing frames in the other description are partially reconstructed using our frame recovery procedure. First, we construct two recovery candidates for each lost pixel in a frame. The first candidate is generated via temporal super-resolution from its predecessor and successor frames in the same view. The second candidate is generated via DIBR fromthe received frame of the same time instance in the other view. Next, we select the best pixel candidates one patch at a time, where an image patch corresponds to a neighborhood of pixels with similar depth values in the 3D scene. Near-optimal source and channel coding rates for each description are selected using a branch-and-bound method, for given transmission bandwidth on each path. Experimental results show that our systemcan outperform a traditional single-description/single-path transmission scheme by up to 5.5 dB in Peak Signal-to-Noise Ratio (PSNR) of the synthesized intermediate view at the client.",
keywords = "Depth-image-based rendering, Free-viewpoints video, Multiple description coding, Wireless multi-path streaming",
author = "Zhi Liu and Gene Cheung and Jacob Chakareski and Yusheng Ji",
year = "2015",
month = "2",
day = "1",
doi = "10.1109/JSTSP.2014.2330332",
language = "English",
volume = "9",
pages = "151--164",
journal = "IEEE Journal on Selected Topics in Signal Processing",
issn = "1932-4553",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "1",

}

TY - JOUR

T1 - Multiple description coding and recovery of free viewpoint video for wireless multi-path streaming

AU - Liu, Zhi

AU - Cheung, Gene

AU - Chakareski, Jacob

AU - Ji, Yusheng

PY - 2015/2/1

Y1 - 2015/2/1

N2 - By transmitting texture and depth videos captured from two nearby camera viewpoints, a client can synthesize via depth-image-based rendering (DIBR) any freely chosen intermediate virtual view of the 3D scene, enhancing the user's perception of depth. During wireless network transmission, burst packet losses can corrupt the transmitted texture and depth videos and degrade the synthesized view quality at the client. In this paper, we propose a multiple description coding system for multi-path transmission of free-viewpoint video, with joint interview and temporal description recovery capability. In particular, we encode separately the even frames of the left view and the odd frames of the right view, and transmit them as one description on one path. The second description comprises the remaining frames in the two views and is transmitted over a second path. If the receiver receives only one description due to burst loss in the other path, the missing frames in the other description are partially reconstructed using our frame recovery procedure. First, we construct two recovery candidates for each lost pixel in a frame. The first candidate is generated via temporal super-resolution from its predecessor and successor frames in the same view. The second candidate is generated via DIBR fromthe received frame of the same time instance in the other view. Next, we select the best pixel candidates one patch at a time, where an image patch corresponds to a neighborhood of pixels with similar depth values in the 3D scene. Near-optimal source and channel coding rates for each description are selected using a branch-and-bound method, for given transmission bandwidth on each path. Experimental results show that our systemcan outperform a traditional single-description/single-path transmission scheme by up to 5.5 dB in Peak Signal-to-Noise Ratio (PSNR) of the synthesized intermediate view at the client.

AB - By transmitting texture and depth videos captured from two nearby camera viewpoints, a client can synthesize via depth-image-based rendering (DIBR) any freely chosen intermediate virtual view of the 3D scene, enhancing the user's perception of depth. During wireless network transmission, burst packet losses can corrupt the transmitted texture and depth videos and degrade the synthesized view quality at the client. In this paper, we propose a multiple description coding system for multi-path transmission of free-viewpoint video, with joint interview and temporal description recovery capability. In particular, we encode separately the even frames of the left view and the odd frames of the right view, and transmit them as one description on one path. The second description comprises the remaining frames in the two views and is transmitted over a second path. If the receiver receives only one description due to burst loss in the other path, the missing frames in the other description are partially reconstructed using our frame recovery procedure. First, we construct two recovery candidates for each lost pixel in a frame. The first candidate is generated via temporal super-resolution from its predecessor and successor frames in the same view. The second candidate is generated via DIBR fromthe received frame of the same time instance in the other view. Next, we select the best pixel candidates one patch at a time, where an image patch corresponds to a neighborhood of pixels with similar depth values in the 3D scene. Near-optimal source and channel coding rates for each description are selected using a branch-and-bound method, for given transmission bandwidth on each path. Experimental results show that our systemcan outperform a traditional single-description/single-path transmission scheme by up to 5.5 dB in Peak Signal-to-Noise Ratio (PSNR) of the synthesized intermediate view at the client.

KW - Depth-image-based rendering

KW - Free-viewpoints video

KW - Multiple description coding

KW - Wireless multi-path streaming

UR - http://www.scopus.com/inward/record.url?scp=84921767996&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84921767996&partnerID=8YFLogxK

U2 - 10.1109/JSTSP.2014.2330332

DO - 10.1109/JSTSP.2014.2330332

M3 - Article

AN - SCOPUS:84921767996

VL - 9

SP - 151

EP - 164

JO - IEEE Journal on Selected Topics in Signal Processing

JF - IEEE Journal on Selected Topics in Signal Processing

SN - 1932-4553

IS - 1

M1 - 6832458

ER -