TY - GEN
T1 - Stacked product quantization for nearest neighbor search on large datasets
AU - Wang, Jun
AU - Li, Zhiyang
AU - Du, Yegang
AU - Qu, Wenyu
N1 - Publisher Copyright:
© 2016 IEEE.
PY - 2016
Y1 - 2016
N2 - High dimensional vector quantization plays an important role in KNN search on large datasets. In recent years, there have a large literature on vector quantization such as product quantization(PQ), optimized product quantization(OPQ), additive quantization (AQ), stacked quantization(SQ). However, these vector quantization faced with large quantization error or low efficiency codebook learning and encoding. In this paper, we propose a new vector quantization method called SPQ which combines the strength of PQ and SQ. On one hand, compared with PQ, we can get a more precise subcodebook in each subspace. On the other hand, we can generate codebook within consuming less time and memory than SQ. Extensive experiments on benchmark datasets demonstrate that SPQ can generate codebook and encoding faster than SQ while maintain the same quantization error. Furthermore we show that SPQ have good scalability, which compare favorably with the sate-of-the-art.
AB - High dimensional vector quantization plays an important role in KNN search on large datasets. In recent years, there have a large literature on vector quantization such as product quantization(PQ), optimized product quantization(OPQ), additive quantization (AQ), stacked quantization(SQ). However, these vector quantization faced with large quantization error or low efficiency codebook learning and encoding. In this paper, we propose a new vector quantization method called SPQ which combines the strength of PQ and SQ. On one hand, compared with PQ, we can get a more precise subcodebook in each subspace. On the other hand, we can generate codebook within consuming less time and memory than SQ. Extensive experiments on benchmark datasets demonstrate that SPQ can generate codebook and encoding faster than SQ while maintain the same quantization error. Furthermore we show that SPQ have good scalability, which compare favorably with the sate-of-the-art.
KW - Big data
KW - High dimensional vector
KW - Nearest neighbor search
KW - Vector quantization
UR - http://www.scopus.com/inward/record.url?scp=85015209061&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85015209061&partnerID=8YFLogxK
U2 - 10.1109/TrustCom.2016.0250
DO - 10.1109/TrustCom.2016.0250
M3 - Conference contribution
AN - SCOPUS:85015209061
T3 - Proceedings - 15th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, 10th IEEE International Conference on Big Data Science and Engineering and 14th IEEE International Symposium on Parallel and Distributed Processing with Applications, IEEE TrustCom/BigDataSE/ISPA 2016
SP - 1621
EP - 1627
BT - Proceedings - 15th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, 10th IEEE International Conference on Big Data Science and Engineering and 14th IEEE International Symposium on Parallel and Distributed Processing with Applications, IEEE TrustCom/BigDataSE/ISPA 2016
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - Joint 15th IEEE International Conference on Trust, Security and Privacy in Computing and Communications, 10th IEEE International Conference on Big Data Science and Engineering and 14th IEEE International Symposium on Parallel and Distributed Processing with Applications, IEEE TrustCom/BigDataSE/ISPA 2016
Y2 - 23 August 2016 through 26 August 2016
ER -