Multiple description coding and recovery of free viewpoint video for wireless multi-path streaming

Zhi Liu, Gene Cheung, Jacob Chakareski, Yusheng Ji

Research output: Contribution to journalArticle

39 Citations (Scopus)

Abstract

By transmitting texture and depth videos captured from two nearby camera viewpoints, a client can synthesize via depth-image-based rendering (DIBR) any freely chosen intermediate virtual view of the 3D scene, enhancing the user's perception of depth. During wireless network transmission, burst packet losses can corrupt the transmitted texture and depth videos and degrade the synthesized view quality at the client. In this paper, we propose a multiple description coding system for multi-path transmission of free-viewpoint video, with joint interview and temporal description recovery capability. In particular, we encode separately the even frames of the left view and the odd frames of the right view, and transmit them as one description on one path. The second description comprises the remaining frames in the two views and is transmitted over a second path. If the receiver receives only one description due to burst loss in the other path, the missing frames in the other description are partially reconstructed using our frame recovery procedure. First, we construct two recovery candidates for each lost pixel in a frame. The first candidate is generated via temporal super-resolution from its predecessor and successor frames in the same view. The second candidate is generated via DIBR fromthe received frame of the same time instance in the other view. Next, we select the best pixel candidates one patch at a time, where an image patch corresponds to a neighborhood of pixels with similar depth values in the 3D scene. Near-optimal source and channel coding rates for each description are selected using a branch-and-bound method, for given transmission bandwidth on each path. Experimental results show that our systemcan outperform a traditional single-description/single-path transmission scheme by up to 5.5 dB in Peak Signal-to-Noise Ratio (PSNR) of the synthesized intermediate view at the client.

Original languageEnglish
Article number6832458
Pages (from-to)151-164
Number of pages14
JournalIEEE Journal on Selected Topics in Signal Processing
Volume9
Issue number1
DOIs
Publication statusPublished - 2015 Feb 1
Externally publishedYes

Keywords

  • Depth-image-based rendering
  • Free-viewpoints video
  • Multiple description coding
  • Wireless multi-path streaming

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Signal Processing

Fingerprint Dive into the research topics of 'Multiple description coding and recovery of free viewpoint video for wireless multi-path streaming'. Together they form a unique fingerprint.

  • Cite this