Stacked product quantization for nearest neighbor search on large datasets

Jun Wang, Zhiyang Li, Yegang Du, Wenyu Qu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

High dimensional vector quantization plays an important role in KNN search on large datasets. In recent years, there have a large literature on vector quantization such as product quantization(PQ), optimized product quantization(OPQ), additive quantization (AQ), stacked quantization(SQ). However, these vector quantization faced with large quantization error or low efficiency codebook learning and encoding. In this paper, we propose a new vector quantization method called SPQ which combines the strength of PQ and SQ. On one hand, compared with PQ, we can get a more precise subcodebook in each subspace. On the other hand, we can generate codebook within consuming less time and memory than SQ. Extensive experiments on benchmark datasets demonstrate that SPQ can generate codebook and encoding faster than SQ while maintain the same quantization error. Furthermore we show that SPQ have good scalability, which compare favorably with the sate-of-the-art.

Original languageEnglish
Title of host publicationProceedings - 15th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, 10th IEEE International Conference on Big Data Science and Engineering and 14th IEEE International Symposium on Parallel and Distributed Processing with Applications, IEEE TrustCom/BigDataSE/ISPA 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1621-1627
Number of pages7
ISBN (Electronic)9781509032051
DOIs
Publication statusPublished - 2016
Externally publishedYes
EventJoint 15th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, 10th IEEE International Conference on Big Data Science and Engineering and 14th IEEE International Symposium on Parallel and Distributed Processing with Applications, IEEE TrustCom/BigDataSE/ISPA 2016 - Tianjin, China
Duration: 2016 Aug 232016 Aug 26

Publication series

NameProceedings - 15th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, 10th IEEE International Conference on Big Data Science and Engineering and 14th IEEE International Symposium on Parallel and Distributed Processing with Applications, IEEE TrustCom/BigDataSE/ISPA 2016

Other

OtherJoint 15th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, 10th IEEE International Conference on Big Data Science and Engineering and 14th IEEE International Symposium on Parallel and Distributed Processing with Applications, IEEE TrustCom/BigDataSE/ISPA 2016
Country/TerritoryChina
CityTianjin
Period16/8/2316/8/26

Keywords

  • Big data
  • High dimensional vector
  • Nearest neighbor search
  • Vector quantization

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Information Systems
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'Stacked product quantization for nearest neighbor search on large datasets'. Together they form a unique fingerprint.

Cite this